# HPTMT Parallel Operators for High Performance Data Science & Data Engineering

arXiv: https://arxiv.org/abs/2108.06001 · HTML: https://mlsanity.com/paper/2108.06001/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 42 references that are in the map.
- First posted on arXiv 13 Aug 2021.

## Builds on (most cited first, 7 of 7)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Horovod: fast and easy distributed deep learning in TensorFlow](https://mlsanity.com/paper/1802.05799/) (2018)
- [Towards Scalable Dataframe Systems](https://mlsanity.com/paper/2001.00888/) (2020)
- [High Performance Data Engineering Everywhere](https://mlsanity.com/paper/2007.09589/) (2020)
- [Big Data Systems Meet Machine Learning Challenges: Towards Big Data Science as a Service](https://mlsanity.com/paper/1709.07493/) (2017)
- [Data Engineering for HPC with Python](https://mlsanity.com/paper/2010.06312/) (2020)
- [HPTMT: Operator-Based Architecture for Scalable High-Performance Data-Intensive Frameworks](https://mlsanity.com/paper/2107.12807/) (2021)

## Abstract (arXiv)

Data-intensive applications are becoming commonplace in all science disciplines. They are comprised of a rich set of sub-domains such as data engineering, deep learning, and machine learning. These applications are built around efficient data abstractions and operators that suit the applications of different domains. Often lack of a clear definition of data structures and operators in the field has led to other implementations that do not work well together. The HPTMT architecture that we proposed recently, identifies a set of data structures, operators, and an execution model for creating rich data applications that links all aspects of data engineering and data science together efficiently. This paper elaborates and illustrates this architecture using an end-to-end application with deep learning and data engineering parts working together.