mlsanityData to 14 Oct 2026

MLlib: Machine Learning in Apache Spark

arXiv 1505.06807 ↗First posted 26 May 2015cs.LG cs.DC cs.MScited by 55

Not on the map. Why?

Abstract (arXiv)

Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 May 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 20; 2023: 10; 2024: 3; 2025: 5; 2026: 3.

55 papers.

Sort:

By area:

  1. 2022Data Debugging with Shapley Importance over End-to-End Machine Learning Pipelinescited by 17
  2. 2021Towards Demystifying Serverless Machine Learning Trainingcited by 16
  3. 2022End-to-end Optimization of Machine Learning Prediction Queriescited by 16
  4. 2022Query Processing on Tensor Computation Runtimescited by 12
  5. 2022Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructurescited by 12
  6. 2021A Survey on Deep Reinforcement Learning for Data Processing and Analyticscited by 10
  7. 2022NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learningcited by 7
  8. 2021Functional Collection Programming with Semi-Ring Dictionariescited by 6
  9. 2021NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learningcited by 5
  10. 2022Optimization for Classical Machine Learning Problems on the GPUcited by 5
  11. 2022An Empirical Evaluation of Flow Based Programming in the Machine Learning Deployment Contextcited by 5
  12. 2021Declarative Machine Learning Systemscited by 4
38 more
  1. 2022A State Transition Model for Mobile Notifications via Survival Analysiscited by 4
  2. 2023Test & Evaluation Best Practices for Machine Learning-Enabled Systemscited by 4
  3. 2023KAXAI: An Integrated Environment for Knowledge Analysis and Explainable AIcited by 4
  4. 2024Intelligent Spark Agents: A Modular LangGraph Framework for Scalable, Visualized, and Enhanced Big Data Machine Learning Workflowscited by 4
  5. 2025Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-trainingcited by 4
  6. 2021Extending Isolation Forest for Anomaly Detection in Big Data via K-Meanscited by 3
  7. 2021Large-Scale News Classification using BERT Language Model: Spark NLP Approachcited by 3
  8. 2021Towards better data discovery and collection with flow-based programmingcited by 3
  9. 2022GCF: Generalized Causal Forest for Heterogeneous Treatment Effect Estimation in Online Marketplacecited by 3
  10. 2022Differential testing for machine learning: an analysis for classification algorithms beyond deep learningcited by 3
  11. 2022Streaming Encoding Algorithms for Scalable Hyperdimensional Computingcited by 3
  12. 2023What are the Machine Learning best practices reported by practitioners on Stack Exchange?cited by 3
  13. 2024Mew: Multiplexed Immunofluorescence Image Analysis through an Efficient Multiplex Networkcited by 3
  14. 2021PyKale: Knowledge-Aware Machine Learning from Multiple Sources in Pythoncited by 2
  15. 2021Decision-forest voting scheme for classification of rare classes in network intrusion detectioncited by 2
  16. 2022Stochastic Gradient Descent without Full Data Shufflecited by 2
  17. 2023Multi-Criteria Client Selection and Scheduling with Fairness Guarantee for Federated Learning Servicecited by 2
  18. 2021Evaluating Deep Learning in SystemML using Layer-wise Adaptive Rate Scaling(LARS) Optimizercited by 1
  19. 2022Fast Distributed k-Means with a Small Number of Roundscited by 1
  20. 2022A comprehensive survey on computational learning methods for analysis of gene expression datacited by 1
  21. 2022Improving the Learnability of Machine Learning APIs by Semi-Automated API Wrappingcited by 1
  22. 2023Safe Collaborative Filteringcited by 1
  23. 2023Towards General and Efficient Online Tuning for Sparkcited by 1
  24. 2023Efficient Hybrid Oversampling and Intelligent Undersampling for Imbalanced Big Data Classificationcited by 1
  25. 2024ASTM :Autonomous Smart Traffic Management System Using Artificial Intelligence CNN and LSTMcited by 1
  26. 2025Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflationcited by 1
  27. 2021Data Analytics and Machine Learning Methods, Techniques and Tool for Model-Driven Engineering of Smart IoT Servicescited by 0
  28. 2021Scalable Predictive Time-Series Analysis of COVID-19: Cases and Fatalitiescited by 0
  29. 2021A Survey on Proactive Customer Care: Enabling Science and Steps to Realize itcited by 0
  30. 2022Semantic of Cloud Computing services for Time Series workflowscited by 0
  31. 2022Distributed data analyticscited by 0
  32. 2022On some studies of Fraud Detection Pipeline and related issues from the scope of Ensemble Learning and Graph-based Learningcited by 0
  33. 2022NumS: Scalable Array Programming for the Cloudcited by 0
  34. 2022Benchmarking Apache Spark and Hadoop MapReduce on Big Data Classificationcited by 0
  35. 2023CMLCompiler: A Unified Compiler for Classical Machine Learningcited by 0
  36. 2023What Causes Exceptions in Machine Learning Applications? Mining Machine Learning-Related Stack Traces on Stack Overflowcited by 0
  37. 2023The Landscape of Modern Machine Learning: A Review of Machine, Distributed and Federated Learningcited by 0
  38. 2025Harmonia: End-to-End RAG Serving Optimizationcited by 0

50 most cited of 55 All 55 (JSON)