mlsanityData to 7 Oct 2026

Offline Reinforcement Learning for Learning to Dispatch for Job Shop Scheduling

arXiv 2409.10589 ↗First posted 16 Sep 2024cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The Job Shop Scheduling Problem (JSSP) is a complex combinatorial optimization problem. While online Reinforcement Learning (RL) has shown promise by quickly finding acceptable solutions for JSSP, it faces key limitations: it requires extensive training interactions from scratch leading to sample inefficiency, cannot leverage existing high-quality solutions from traditional methods like Constraint Programming (CP), and require simulated environments to train in, which are impracticable to build for complex scheduling environments. We introduce Offline Learned Dispatching (Offline-LD), an offline reinforcement learning approach for JSSP, which addresses these limitations by learning from historical scheduling data. Our approach is motivated by scenarios where historical scheduling data and expert solutions are available or scenarios where online training of RL approaches with simulated environments is impracticable. Offline-LD introduces maskable variants of two Q-learning methods, namely, Maskable Quantile Regression DQN (mQRDQN) and discrete maskable Soft Actor-Critic (d-mSAC), that are able to learn from historical data, through Conservative Q-Learning (CQL). Moreover, we present a novel entropy bonus modification for d-mSAC, for maskable action spaces. Moreover, we introduce a novel reward normalization method for JSSP in an offline RL setting. Our experiments demonstrate that Offline-LD outperforms online RL on both generated and benchmark instances when trained on only 100 solutions generated by CP. Notably, introducing noise to the expert dataset yields comparable or superior results to using the expert dataset, with the same amount of instances, a promising finding for real-world applications, where data is inherently noisy and imperfect.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 51 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

28 of 51 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets6 Oct
  2. 2023Survival Instinct in Offline Reinforcement Learning5 Jun
  3. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018How Powerful are Graph Neural Networks?cited by 4,355
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2017Deep Reinforcement Learning that Matterscited by 727
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  6. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  8. 2021Implicit Behavioral Cloningcited by 442
  9. 2021Combinatorial optimization and reasoning with graph neural networkscited by 267
  10. 2020Reinforcement Learning for Combinatorial Optimization: A Surveycited by 235
  11. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  12. 2020Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement Learningcited by 139
8 more
  1. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  2. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  3. 2020Combining Reinforcement Learning and Constraint Programming for Combinatorial Optimizationcited by 63
  4. 2023Flexible Job Shop Scheduling via Dual Attention Network Based Reinforcement Learningcited by 20
  5. 2024Graph Neural Networks for Job Shop Scheduling Problems: A Surveycited by 15
  6. 2023An End-to-End Reinforcement Learning Approach for Job-Shop Scheduling Problems Based on Constraint Programmingcited by 11
  7. 2023Job Shop Scheduling Benchmark: Environments and Instances for Learning and Non-learning Methodscited by 11
  8. 2022ML4CO-KIDA: Knowledge Inheritance in Dataset Aggregationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Solutionscited by 0
  2. 2026TempoNet: Slack-Quantized Transformer-Guided Reinforcement Scheduler for Adaptive Deadline-Centric Real-Time Dispatchscited by 0
  3. 2026Teacher-Aware Evolution of Heuristic Programs from Learned Optimization Policiescited by 0
  4. 2026Scheduling That Speaks: An Interpretable Programmatic Reinforcement Learning Frameworkcited by 0
  5. 2026Bridging the Sim-to-Real Gap in Reinforcement Learning-Based Industrial Dispatching through Execution Semanticscited by 0
  6. 2026PORL: Pretrained Offline Reinforcement Learning for the Job Shop Scheduling Problemcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline reinforcement learning for job-shop scheduling problems
  2. 2022Launchpad: Learning to Schedule Using Offline and Online RL Methods
  3. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
  4. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
  5. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL
  6. 2020MOReL : Model-Based Offline Reinforcement Learning
  7. 2020Hyperparameter Selection for Offline Reinforcement Learning
  8. 2020MOPO: Model-based Offline Policy Optimization