# Operon: Incremental Construction of Ragged Data via Named Dimensions

arXiv: https://arxiv.org/abs/2511.16080 · HTML: https://mlsanity.com/paper/2511.16080/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 41 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Builds on (most cited first, 13 of 13)

- [Scaling Large Language Model-based Multi-Agent Collaboration](https://mlsanity.com/paper/2406.07155/) (2024)
- [AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2504.00587/) (2025)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum](https://mlsanity.com/paper/2405.13226/) (2024)
- [SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning](https://mlsanity.com/paper/2306.03491/) (2023)
- [The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Padding](https://mlsanity.com/paper/2110.10221/) (2021)
- [Open World Object Detection: A Survey](https://mlsanity.com/paper/2410.11301/) (2024)
- [Named Tensor Notation](https://mlsanity.com/paper/2102.13196/) (2021)
- [Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming](https://mlsanity.com/paper/2104.05372/) (2021)
- [Orchestrating Human-AI Teams: The Manager Agent as a Unifying Research Challenge](https://mlsanity.com/paper/2510.02557/) (2025)
- [Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluation](https://mlsanity.com/paper/2408.01363/) (2024)
- [A Practical Approach for Building Production-Grade Conversational Agents with Workflow Graphs](https://mlsanity.com/paper/2505.23006/) (2025)
- [AwkwardForth: accelerating Uproot with an internal DSL](https://mlsanity.com/paper/2102.13516/) (2021)

## Abstract (arXiv)

Modern data processing workflows frequently encounter ragged data: collections with variable-length elements that arise naturally in domains like natural language processing, scientific measurements, and autonomous AI agents. Existing workflow engines lack native support for tracking the shapes and dependencies inherent to ragged data, forcing users to manage complex indexing and dependency bookkeeping manually. We present Operon, a Rust-based workflow engine that addresses these challenges through a novel formalism of named dimensions with explicit dependency relations. Operon provides a domain-specific language where users declare pipelines with dimension annotations that are statically verified for correctness, while the runtime system dynamically schedules tasks as data shapes are incrementally discovered during execution. We formalize the mathematical foundation for reasoning about partial shapes and prove that Operon's incremental construction algorithm guarantees deterministic and confluent execution in parallel settings. The system's explicit modeling of partially-known states enables robust persistence and recovery mechanisms, while its per-task multi-queue architecture achieves efficient parallelism across heterogeneous task types. Empirical evaluation demonstrates that Operon outperforms an existing workflow engine with 14.94x baseline overhead reduction while maintaining near-linear end-to-end output rates as workloads scale, making it particularly suitable for large-scale data generation pipelines in machine learning applications.