# High Performance Dataframes from Parallel Processing Patterns

arXiv: https://arxiv.org/abs/2209.06146 · HTML: https://mlsanity.com/paper/2209.06146/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2022.

## Built on by (most cited first, 4 of 4)

- [In-depth Analysis On Parallel Processing Patterns for High-Performance Dataframes](https://mlsanity.com/paper/2307.01394/) (2023)
- [AAFLOW: Scalable Patterns for Agentic AI Workflows](https://mlsanity.com/paper/2605.02162/) (2026)
- [Mathematical Computation on High-dimensional Data via Array Programming and Parallel Acceleration](https://mlsanity.com/paper/2506.22929/) (2025)
- [OpRAG: A Resource-Deterministic Runtime for GPU-Backed Multi-Stage RAG Workflows](https://mlsanity.com/paper/2608.08340/) (2026)

## Abstract (arXiv)

The data science community today has embraced the concept of Dataframes as the de facto standard for data representation and manipulation. Ease of use, massive operator coverage, and popularization of R and Python languages have heavily influenced this transformation. However, most widely used serial Dataframes today (R, pandas) experience performance limitations even while working on even moderately large data sets. We believe that there is plenty of room for improvement by investigating the generic distributed patterns of dataframe operators. In this paper, we propose a framework that lays the foundation for building high performance distributed-memory parallel dataframe systems based on these parallel processing patterns. We also present Cylon, as a reference runtime implementation. We demonstrate how this framework has enabled Cylon achieving scalable high performance. We also underline the flexibility of the proposed API and the extensibility of the framework on different hardware. To the best of our knowledge, Cylon is the first and only distributed-memory parallel dataframe system available today.