# MLlib: Machine Learning in Apache Spark

arXiv: https://arxiv.org/abs/1505.06807 · HTML: https://mlsanity.com/paper/1505.06807/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2015.

## Built on by (most cited first, 30 of 55)

- [Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelines](https://mlsanity.com/paper/2204.11131/) (2022)
- [Towards Demystifying Serverless Machine Learning Training](https://mlsanity.com/paper/2105.07806/) (2021)
- [End-to-end Optimization of Machine Learning Prediction Queries](https://mlsanity.com/paper/2206.00136/) (2022)
- [Query Processing on Tensor Computation Runtimes](https://mlsanity.com/paper/2203.01877/) (2022)
- [Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructures](https://mlsanity.com/paper/2205.04713/) (2022)
- [A Survey on Deep Reinforcement Learning for Data Processing and Analytics](https://mlsanity.com/paper/2108.04526/) (2021)
- [NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learning](https://mlsanity.com/paper/2210.00973/) (2022)
- [Functional Collection Programming with Semi-Ring Dictionaries](https://mlsanity.com/paper/2103.06376/) (2021)
- [NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning](https://mlsanity.com/paper/2111.13984/) (2021)
- [Optimization for Classical Machine Learning Problems on the GPU](https://mlsanity.com/paper/2203.16340/) (2022)
- [An Empirical Evaluation of Flow Based Programming in the Machine Learning Deployment Context](https://mlsanity.com/paper/2204.12781/) (2022)
- [Declarative Machine Learning Systems](https://mlsanity.com/paper/2107.08148/) (2021)
- [A State Transition Model for Mobile Notifications via Survival Analysis](https://mlsanity.com/paper/2207.03099/) (2022)
- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [KAXAI: An Integrated Environment for Knowledge Analysis and Explainable AI](https://mlsanity.com/paper/2401.00193/) (2023)
- [Intelligent Spark Agents: A Modular LangGraph Framework for Scalable, Visualized, and Enhanced Big Data Machine Learning Workflows](https://mlsanity.com/paper/2412.01490/) (2024)
- [Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-training](https://mlsanity.com/paper/2502.16802/) (2025)
- [Extending Isolation Forest for Anomaly Detection in Big Data via K-Means](https://mlsanity.com/paper/2104.13190/) (2021)
- [Large-Scale News Classification using BERT Language Model: Spark NLP Approach](https://mlsanity.com/paper/2107.06785/) (2021)
- [Towards better data discovery and collection with flow-based programming](https://mlsanity.com/paper/2108.04105/) (2021)
- [GCF: Generalized Causal Forest for Heterogeneous Treatment Effect Estimation in Online Marketplace](https://mlsanity.com/paper/2203.10975/) (2022)
- [Differential testing for machine learning: an analysis for classification algorithms beyond deep learning](https://mlsanity.com/paper/2207.11976/) (2022)
- [Streaming Encoding Algorithms for Scalable Hyperdimensional Computing](https://mlsanity.com/paper/2209.09868/) (2022)
- [What are the Machine Learning best practices reported by practitioners on Stack Exchange?](https://mlsanity.com/paper/2301.10516/) (2023)
- [Mew: Multiplexed Immunofluorescence Image Analysis through an Efficient Multiplex Network](https://mlsanity.com/paper/2407.17857/) (2024)
- [PyKale: Knowledge-Aware Machine Learning from Multiple Sources in Python](https://mlsanity.com/paper/2106.09756/) (2021)
- [Decision-forest voting scheme for classification of rare classes in network intrusion detection](https://mlsanity.com/paper/2107.11862/) (2021)
- [Stochastic Gradient Descent without Full Data Shuffle](https://mlsanity.com/paper/2206.05830/) (2022)
- [Multi-Criteria Client Selection and Scheduling with Fairness Guarantee for Federated Learning Service](https://mlsanity.com/paper/2312.14941/) (2023)
- [Evaluating Deep Learning in SystemML using Layer-wise Adaptive Rate Scaling(LARS) Optimizer](https://mlsanity.com/paper/2102.03018/) (2021)

## Abstract (arXiv)

Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.