DataJoint 2.0 Release Plan

[dimitri-yatsenko]

As we approach the final stages of development, we are excited to announce DataJoint 2.0, our first major release designed to align with the official DataJoint Specs 2.0

Target release date: Oct 2025.

Tested Python Versions: Python 3.10-3.13

---

The centerpiece of this release is the introduction of Object-Augmented Schemas. We are also delivering a fully modernized framework for project configuration, data handling, and new scientific datatype support.

Compatibility Note: While we aim for data compatibility (meaning no data migration should be required), this release will introduce several backward-incompatible API features. Please plan to update your code to leverage the new functionalities.

---

Object-Augmented Schemas

DataJoint 2.0 fundamentally revises how relational tables are combined with large-scale scientific data objects. This new paradigm allows for more flexible and efficient handling of complex data types directly within the data pipeline.

• Core Implementation: Foundational support for Object-Augmented Schemas. FEAT: Object-augmented schemas -- Object Type #1259
• Custom Data Type Plugins: An extensible architecture for user-defined data types. FEAT: Custom Types 2.0 #1260
  • <dj_blob>: A plugin for backward-compatible DataJoint blobs. FEAT: dj_blob type plugin for DataJoint 2.0 #1262
  • <dj_zarr>: A plugin for general Zarr 3.0 array support FEAT: Implement <dj_zarr> Custom Type Adaptor #1263
  • <multiphoton>: plugin for handling compressed, photon-limited movies (based on zarr)

Data Export & Import

• Diagram-Based Restriction: Use data diagrams to visually define and restrict data for queries (FEAT: Applying Restrictions to a Diagram #865).
• Workflow Publishing: Streamlined, diagram-based tools for exporting and publishing datasets.
• Database Restore: Functionality to restore a database state from an export.

AutoPopulate 2.0. #1258

This is a redesigned, scalable job management engine to replace the existing autopopulate framework.

• Per-Table Job Queues: Each computed table gets its own dedicated jobs table, accessible via Table.jobs.
• Direct Key Addressing: Jobs are identified by the primary key of the main table, not by hashes.
• Improved Job Lifecycle Management: A job queue with statuses like pending, reserved, and error, and an explicit refresh() method to update the queue.
• Job Prioritization: A new priority field allows for controlling the execution order of jobs.
• Enhanced Performance: Reduces database strain by eliminating frequent and expensive key_source queries from multiple workers.

Project Configuration & Extensibility

• Unified Management: A single system for handling secrets, config files, and environment variables. (IMPR: Modernize Project Configuration Management #1257)
• Modern Plugin Architecture: A new framework for enhancing and extending core DataJoint functionality. (IMPR: Upgrade plugin architecture #1251)

Schema Definition & Data Types

• Scientific Datatype Aliases: Support for common types like float32, float64, uint8, int8, uint16, int16, uint32, int32, uint64, and int64. FEAT: Core attributes types 2.0 #1261

Version 2.1 (Post 2.0 Release)

The following features are planned for a subsequent release shortly after 2.0, with a possibility of being included in the main release.

• PostgreSQL Support
• Native JSON Datatype Support (2.0) -- Postgres compatible