# A Tensor Compiler for Unified Machine Learning Prediction Serving

arXiv: https://arxiv.org/abs/2010.04804 · HTML: https://mlsanity.com/paper/2010.04804/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2020.

## Built on by (most cited first, 20 of 20)

- [Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture](https://mlsanity.com/paper/2103.03330/) (2021)
- [Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelines](https://mlsanity.com/paper/2204.11131/) (2022)
- [End-to-end Optimization of Machine Learning Prediction Queries](https://mlsanity.com/paper/2206.00136/) (2022)
- [Query Processing on Tensor Computation Runtimes](https://mlsanity.com/paper/2203.01877/) (2022)
- [Share the Tensor Tea: How Databases can Leverage the Machine Learning Ecosystem](https://mlsanity.com/paper/2209.04579/) (2022)
- [TOD: GPU-accelerated Outlier Detection via Tensor Operations](https://mlsanity.com/paper/2110.14007/) (2021)
- [MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training](https://mlsanity.com/paper/2407.12117/) (2024)
- [FLInt: Exploiting Floating Point Enabled Integer Arithmetic for Efficient Random Forest Inference](https://mlsanity.com/paper/2209.04181/) (2022)
- [The Tensor Data Platform: Towards an AI-centric Database System](https://mlsanity.com/paper/2211.02753/) (2022)
- [A Comparison of Decision Forest Inference Platforms from A Database Perspective](https://mlsanity.com/paper/2302.04430/) (2023)
- [Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelines](https://mlsanity.com/paper/2405.11191/) (2024)
- [Fast Inference of Tree Ensembles on ARM Devices](https://mlsanity.com/paper/2305.08579/) (2023)
- [OptiLeak: Efficient Prompt Reconstruction via Reinforcement Learning in Multi-tenant LLM Services](https://mlsanity.com/paper/2602.20595/) (2026)
- [CMLCompiler: A Unified Compiler for Classical Machine Learning](https://mlsanity.com/paper/2301.13441/) (2023)
- [Register Your Forests: Decision Tree Ensemble Optimization by Explicit CPU Register Allocation](https://mlsanity.com/paper/2404.06846/) (2024)
- [TREE: Tree Regularization for Efficient Execution](https://mlsanity.com/paper/2406.12531/) (2024)
- [DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures](https://mlsanity.com/paper/2511.15503/) (2025)
- [Vectorized Dynamic Histograms for Sparse Oblique Forests](https://mlsanity.com/paper/2603.00326/) (2026)
- [Forge-UGC: FX optimization and register-graph engine for universal graph compiler](https://mlsanity.com/paper/2604.16498/) (2026)
- [TreeWalker: Partial Evaluation for Grouped Tree-Ensemble Inference](https://mlsanity.com/paper/2610.03939/) (2026)

## Abstract (arXiv)

Machine Learning (ML) adoption in the enterprise requires simpler and more efficient software infrastructure---the bespoke solutions typical in large web companies are simply untenable. Model scoring, the process of obtaining predictions from a trained model over new data, is a primary contributor to infrastructure complexity and cost as models are trained once but used many times. In this paper we propose HUMMINGBIRD, a novel approach to model scoring, which compiles featurization operators and traditional ML models (e.g., decision trees) into a small set of tensor operations. This approach inherently reduces infrastructure complexity and directly leverages existing investments in Neural Network compilers and runtimes to generate efficient computations for both CPU and hardware accelerators. Our performance results are intriguing: despite replacing imperative computations (e.g., tree traversals) with tensor computation abstractions, HUMMINGBIRD is competitive and often outperforms hand-crafted kernels on micro-benchmarks on both CPU and GPU, while enabling seamless end-to-end acceleration of ML pipelines. We have released HUMMINGBIRD as open source.