# Deep Inventory Management

arXiv: https://arxiv.org/abs/2210.03137 · HTML: https://mlsanity.com/paper/2210.03137/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 57 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [A Multi-Horizon Quantile Recurrent Forecaster](https://mlsanity.com/paper/1711.11053/) (2017)
- [ChainQueen: A Real-Time Differentiable Physical Simulator for Soft Robotics](https://mlsanity.com/paper/1810.01054/) (2018)
- [Do Differentiable Simulators Give Better Policy Gradients?](https://mlsanity.com/paper/2202.00817/) (2022)
- [Simple random search provides a competitive approach to reinforcement learning](https://mlsanity.com/paper/1803.07055/) (2018)
- [MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention](https://mlsanity.com/paper/2009.14799/) (2020)
- [Sample-Efficient Reinforcement Learning in the Presence of Exogenous Information](https://mlsanity.com/paper/2206.04282/) (2022)
- [Hindsight Learning for MDPs with Exogenous Inputs](https://mlsanity.com/paper/2207.06272/) (2022)
- [Applying Deep Learning to the Newsvendor Problem](https://mlsanity.com/paper/1607.02177/) (2016)
- [ORL: Reinforcement Learning Benchmarks for Online Stochastic Optimization Problems](https://mlsanity.com/paper/1911.10641/) (2019)
- [Deep Generative Quantile-Copula Models for Probabilistic Forecasting](https://mlsanity.com/paper/1907.10697/) (2019)
- [Sparsity in Partially Controllable Linear Systems](https://mlsanity.com/paper/2110.06150/) (2021)
- [MQRetNN: Multi-Horizon Time Series Forecasting with Retrieval Augmentation](https://mlsanity.com/paper/2207.10517/) (2022)

## Built on by (most cited first, 23 of 23)

- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [Policy Optimization for Continuous Reinforcement Learning](https://mlsanity.com/paper/2305.18901/) (2023)
- [An Analysis of Multi-Agent Reinforcement Learning for Decentralized Inventory Control Systems](https://mlsanity.com/paper/2307.11432/) (2023)
- [AI Agents for Inventory Control: Human-LLM-OR Complementarity](https://mlsanity.com/paper/2602.12631/) (2026)
- [What model does MuZero learn?](https://mlsanity.com/paper/2306.00840/) (2023)
- [Learning an Inventory Control Policy with General Inventory Arrival Dynamics](https://mlsanity.com/paper/2310.17168/) (2023)
- [Ask, Clarify, Optimize: Human-LLM Agent Collaboration for Smarter Inventory Control](https://mlsanity.com/paper/2601.00121/) (2025)
- [DeepStock: Reinforcement Learning with Policy Regularizations for Inventory Management](https://mlsanity.com/paper/2603.19621/) (2026)
- [Differentiable Discrete Event Simulation for Queuing Network Control](https://mlsanity.com/paper/2409.03740/) (2024)
- [Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2409.14557/) (2024)
- [The Fallacy of Minimizing Cumulative Regret in the Sequential Task Setting](https://mlsanity.com/paper/2403.10946/) (2024)
- [Speeding up Policy Simulation in Supply Chain RL](https://mlsanity.com/paper/2406.01939/) (2024)
- [gym-invmgmt: An Open Benchmarking Framework for Inventory Management Methods](https://mlsanity.com/paper/2605.11355/) (2026)
- [Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients](https://mlsanity.com/paper/2605.14297/) (2026)
- [Optimal Hidden-Target Learning for Online Inventory Optimization on General Convex Sets](https://mlsanity.com/paper/2606.14679/) (2026)
- [OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Items](https://mlsanity.com/paper/2609.01933/) (2026)
- [Contextual Bandits for Evaluating and Improving Inventory Control Policies](https://mlsanity.com/paper/2310.16096/) (2023)
- [Pivoting Retail Supply Chain with Deep Generative Techniques: Taxonomy, Survey and Insights](https://mlsanity.com/paper/2403.00861/) (2024)
- [Generative Probabilistic Planning for Optimizing Supply Chain Networks](https://mlsanity.com/paper/2404.07511/) (2024)
- [Zero-shot Generalization in Inventory Management: Train, then Estimate and Decide](https://mlsanity.com/paper/2411.00515/) (2024)
- [The Spoils of Algorithmic Collusion: Profit Allocation Among Asymmetric Firms](https://mlsanity.com/paper/2501.07178/) (2025)
- [Is Pure Exploitation Sufficient in Exogenous MDPs with Linear Function Approximation?](https://mlsanity.com/paper/2601.20694/) (2026)
- [COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Management](https://mlsanity.com/paper/2603.02396/) (2026)

## Abstract (arXiv)

This work provides a Deep Reinforcement Learning approach to solving a periodic review inventory control system with stochastic vendor lead times, lost sales, correlated demand, and price matching. While this dynamic program has historically been considered intractable, our results show that several policy learning approaches are competitive with or outperform classical methods. In order to train these algorithms, we develop novel techniques to convert historical data into a simulator. On the theoretical side, we present learnability results on a subclass of inventory control problems, where we provide a provable reduction of the reinforcement learning problem to that of supervised learning. On the algorithmic side, we present a model-based reinforcement learning procedure (Direct Backprop) to solve the periodic review inventory control problem by constructing a differentiable simulator. Under a variety of metrics Direct Backprop outperforms model-free RL and newsvendor baselines, in both simulations and real-world deployments.