# OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Items

arXiv: https://arxiv.org/abs/2609.01933 · HTML: https://mlsanity.com/paper/2609.01933/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Gradient Estimation Using Stochastic Computation Graphs](https://mlsanity.com/paper/1506.05254/) (2015)
- [Parallelizing the dual revised simplex method](https://mlsanity.com/paper/1503.01889/) (2015)
- [The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise](https://mlsanity.com/paper/2401.07844/) (2024)
- [Deep Inventory Management](https://mlsanity.com/paper/2210.03137/) (2022)
- [Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design](https://mlsanity.com/paper/2301.13734/) (2023)
- [Beyond Pessimism: Offline Learning in KL-regularized Games](https://mlsanity.com/paper/2604.06738/) (2026)
- [Efficient Multi-Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/2408.08706/) (2024)
- [VC Theory for Inventory Policies](https://mlsanity.com/paper/2404.11509/) (2024)
- [Doubly Optimal Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/2410.02226/) (2024)
- [AI Agents for Inventory Control: Human-LLM-OR Complementarity](https://mlsanity.com/paper/2602.12631/) (2026)
- [Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2604.09855/) (2026)
- [Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning](https://mlsanity.com/paper/2410.05655/) (2024)
- [Offline Two-Player Zero-Sum Markov Games with KL Regularization](https://mlsanity.com/paper/2605.13025/) (2026)
- [DeepStock: Reinforcement Learning with Policy Regularizations for Inventory Management](https://mlsanity.com/paper/2603.19621/) (2026)
- [Pessimism-Free Offline Learning in General-Sum Games via KL Regularization](https://mlsanity.com/paper/2605.00264/) (2026)
- [Fast Rates in α-Potential Games via Regularized Mirror Descent](https://mlsanity.com/paper/2605.00268/) (2026)
- [Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning](https://mlsanity.com/paper/2605.07333/) (2026)
- [Optimal Pricing of Information](https://mlsanity.com/paper/2102.13289/) (2021)
- [Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective](https://mlsanity.com/paper/2605.09044/) (2026)
- [Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations](https://mlsanity.com/paper/2607.05863/) (2026)
- [Convergence of Two-Timescale Markovian Stochastic Approximations with Applications in Reinforcement Learning](https://mlsanity.com/paper/2605.31172/) (2026)
- [Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluation](https://mlsanity.com/paper/2608.24146/) (2026)
- [gym-invmgmt: An Open Benchmarking Framework for Inventory Management Methods](https://mlsanity.com/paper/2605.11355/) (2026)
- [Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients](https://mlsanity.com/paper/2605.14297/) (2026)

## Built on by (most cited first, 1 of 1)

- [Pessimistic Minimax Learning for Public-Private Information Games under Unilateral Coverage](https://mlsanity.com/paper/2610.04997/) (2026)

## Abstract (arXiv)

Modern supply chain operations can require coordinating replenishment across thousands of heterogeneous items under correlated stochastic demand, heterogeneous lead times, and shared fixed ordering costs, yielding observation spaces exceeding 104 dimensions. At this scale, rolling-horizon stochastic mixed-integer linear programs (MILPs) become prohibitively slow, while standard reinforcement learning (RL) methods face increasingly challenging credit assignment in high-dimensional action spaces. We introduce OR-Transformer, a deep reinforcement learning framework for joint replenishment under stochastic demand, with an item-permutation-equivariant Transformer architecture and pathwise-gradient training through the inventory dynamics. Across problem sizes up to 1,024 inventory items, OR-Transformer increasingly outperforms learning-based and rolling-horizon MILP baselines as scale grows. It also reduces online decision-making time by over 4 million times relative to MILP solvers, enabling real-time, large-scale deep RL in supply chain operations.