mlsanityData to 14 Oct 2026

Learning Scheduling Algorithms for Data Processing Clusters

arXiv 1810.01963 ↗First posted 3 Oct 2018cs.LG stat.MLcited by 134

Not on the map. Why?

Abstract (arXiv)

Efficiently scheduling data processing jobs on distributed compute clusters requires complex algorithms. Current systems, however, use simple generalized heuristics and ignore workload characteristics, since developing and tuning a scheduling policy for each workload is infeasible. In this paper, we show that modern machine learning techniques can generate highly-efficient policies automatically. Decima uses reinforcement learning (RL) and neural networks to learn workload-specific scheduling algorithms without any human instruction beyond a high-level objective such as minimizing average job completion time. Off-the-shelf RL techniques, however, cannot handle the complexity and scale of the scheduling problem. To build Decima, we had to develop new representations for jobs' dependency graphs, design scalable RL models, and invent RL training methods for dealing with continuous stochastic job arrivals. Our prototype integration with Spark on a 25-node cluster shows that Decima improves the average job completion time over hand-tuned scheduling heuristics by at least 21%, achieving up to 2x improvement during periods of high cluster load.

Key facts. Cited by 134 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 33; 2022: 24; 2023: 16; 2024: 15; 2025: 22; 2026: 24.

134 papers.

Sort:

By area:

  1. 2024NetLLM: Adapting Large Language Models for Networkingcited by 78
  2. 2023Robust Scheduling with GFlowNetscited by 69
  3. 2025Barbarians at the Gate: How AI is Upending Systems Researchcited by 54
  4. 2021A Survey of Machine Learning for Computer Architecture and Systemscited by 53
  5. 2021Efficient Device Scheduling with Multi-Job Federated Learningcited by 29
  6. 2021A Bi-Level Framework for Learning to Solve Combinatorial Optimization on Graphscited by 26
  7. 2022Hindsight Learning for MDPs with Exogenous Inputscited by 26
  8. 2024Self-Guiding Exploration for Combinatorial Problemscited by 22
  9. 2023Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunitiescited by 21
  10. 2022Multi-Job Intelligent Scheduling with Cross-Device Federated Learningcited by 20
  11. 2022Scalable Verification of GNN-based Job Schedulerscited by 19
  12. 2022Zero-Shot Cost Models for Out-of-the-box Learned Cost Predictioncited by 16
38 more
  1. 2025Glia: A Human-Inspired AI for Automated Systems Design and Optimizationcited by 16
  2. 2021MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Corescited by 15
  3. 2022A Survey on Influence Maximization: From an ML-Based Combinatorial Optimizationcited by 15
  4. 2023AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challengescited by 15
  5. 2021Contingency-Aware Influence Maximization: A Reinforcement Learning Approachcited by 13
  6. 2022Autothrottle: A Practical Bi-Level Approach to Resource Management for SLO-Targeted Microservicescited by 12
  7. 2021FLASH: Fast Neural Architecture Search with Hardware Optimizationcited by 11
  8. 2021Deep Reinforcement Agent for Scheduling in HPCcited by 10
  9. 2021A Survey on Deep Reinforcement Learning for Data Processing and Analyticscited by 10
  10. 2021SOLO: Search Online, Learn Offline for Combinatorial Optimization Problemscited by 9
  11. 2021One Model to Rule them All: Towards Zero-Shot Learning for Databasescited by 9
  12. 2021Accelerating Deep Learning Inference via Learned Cachescited by 8
  13. 2021Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusterscited by 7
  14. 2021OnSlicing: Online End-to-End Network Slicing with Reinforcement Learningcited by 7
  15. 2022Learning to generalize Dispatching rules on the Job Shop Schedulingcited by 7
  16. 2023RESPECT: Reinforcement Learning based Edge Scheduling on Pipelined Coral Edge TPUscited by 7
  17. 2024Efficient Reinforcement Learning for Routing Jobs in Heterogeneous Queueing Systemscited by 7
  18. 2021IGNNITION: Bridging the Gap Between Graph Neural Networks and Networking Systemscited by 6
  19. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 6
  20. 2023Online Reinforcement Learning in Non-Stationary Context-Driven Environmentscited by 6
  21. 2025Graph Neural Networks for Databases: A Surveycited by 6
  22. 2021A Graph Policy Network Approach for Volt-Var Control in Power Distribution Systemscited by 5
  23. 2024Performance of NPG in Countable State-Space Average-Cost RLcited by 5
  24. 2021Machine Learning (ML)-Centric Resource Management in Cloud Computing: A Review and Future Directionscited by 4
  25. 2021AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruningcited by 4
  26. 2021ENERO: Efficient Real-Time WAN Routing Optimization with Deep Reinforcement Learningcited by 4
  27. 2021Learning for Robust Combinatorial Optimization: Algorithm and Applicationcited by 4
  28. 2023Accelerating Exact Combinatorial Optimization via RL-based Initialization -- A Case Study in Schedulingcited by 4
  29. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learningcited by 4
  30. 2023Predict-Then-Optimize by Proxy: Learning Joint Models of Prediction and Optimizationcited by 4
  31. 2024Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMscited by 4
  32. 2026Larch: Learned Query Optimization for Semantic Predicatescited by 4
  33. 2021Reinforced Workload Distribution Fairnesscited by 3
  34. 2023Tree Search in DAG Space with Model-based Reinforcement Learning for Causal Discoverycited by 3
  35. 2024Learning Joint Models of Prediction and Optimizationcited by 3
  36. 2025FlowletFormer: Network Behavioral Semantic Aware Pre-training Model for Traffic Classificationcited by 3
  37. 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 3
  38. 2021GCNScheduler: Scheduling Distributed Computing Applications using Graph Convolutional Networkscited by 2

50 most cited of 134 All 134 (JSON)