# Towards Scalable Dataframe Systems

arXiv: https://arxiv.org/abs/2001.00888 · HTML: https://mlsanity.com/paper/2001.00888/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jan 2020.

## Built on by (most cited first, 17 of 17)

- [Balsa: Learning a Query Optimizer Without Expert Demonstrations](https://mlsanity.com/paper/2201.01441/) (2022)
- [Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelines](https://mlsanity.com/paper/2204.11131/) (2022)
- [Query Processing on Tensor Computation Runtimes](https://mlsanity.com/paper/2203.01877/) (2022)
- [LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology](https://mlsanity.com/paper/2509.13978/) (2025)
- [Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie](https://mlsanity.com/paper/2404.13682/) (2024)
- [In-depth Analysis On Parallel Processing Patterns for High-Performance Dataframes](https://mlsanity.com/paper/2307.01394/) (2023)
- [HPTMT: Operator-Based Architecture for Scalable High-Performance Data-Intensive Frameworks](https://mlsanity.com/paper/2107.12807/) (2021)
- [Bauplan: zero-copy, scale-up FaaS for data pipelines](https://mlsanity.com/paper/2410.17465/) (2024)
- [AAFLOW: Scalable Patterns for Agentic AI Workflows](https://mlsanity.com/paper/2605.02162/) (2026)
- [[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows](https://mlsanity.com/paper/2607.10987/) (2026)
- [Transactional Python for Durable Machine Learning: Vision, Challenges, and Feasibility](https://mlsanity.com/paper/2305.08770/) (2023)
- [stratum: A System Infrastructure for Massive Agent-Centric ML Workloads](https://mlsanity.com/paper/2603.03589/) (2026)
- [OpRAG: A Resource-Deterministic Runtime for GPU-Backed Multi-Stage RAG Workflows](https://mlsanity.com/paper/2608.08340/) (2026)
- [HPTMT Parallel Operators for High Performance Data Science & Data Engineering](https://mlsanity.com/paper/2108.06001/) (2021)
- [NumS: Scalable Array Programming for the Cloud](https://mlsanity.com/paper/2206.14276/) (2022)
- [Towards Multifaceted Human-Centered AI](https://mlsanity.com/paper/2301.03656/) (2023)
- [PandasCorpus: A Resource of Real-World Pandas Workflows and Usage Patterns](https://mlsanity.com/paper/2608.14742/) (2026)

## Abstract (arXiv)

Dataframes are a popular abstraction to represent, prepare, and analyze data. Despite the remarkable success of dataframe libraries in Rand Python, dataframes face performance issues even on moderately large datasets. Moreover, there is significant ambiguity regarding dataframe semantics. In this paper we lay out a vision and roadmap for scalable dataframe systems. To demonstrate the potential in this area, we report on our experience building MODIN, a scaled-up implementation of the most widely-used and complex dataframe API today, Python's pandas. With pandas as a reference, we propose a simple data model and algebra for dataframes to ground discussion in the field. Given this foundation, we lay out an agenda of open research opportunities where the distinct features of dataframes will require extending the state of the art in many dimensions of data management. We discuss the implications of signature data-frame features including flexible schemas, ordering, row/column equivalence, and data/metadata fluidity, as well as the piecemeal, trial-and-error-based approach to interacting with dataframes.