# Deep Reinforcement Learning for Inventory Networks: Toward Reliable Policy Optimization

arXiv: https://arxiv.org/abs/2306.11246 · HTML: https://mlsanity.com/paper/2306.11246/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2023.

## Built on by (most cited first, 13 of 13)

- [VC Theory for Inventory Policies](https://mlsanity.com/paper/2404.11509/) (2024)
- [Neural Coordination and Capacity Control for Inventory Management](https://mlsanity.com/paper/2410.02817/) (2024)
- [Learning an Inventory Control Policy with General Inventory Arrival Dynamics](https://mlsanity.com/paper/2310.17168/) (2023)
- [Ask, Clarify, Optimize: Human-LLM Agent Collaboration for Smarter Inventory Control](https://mlsanity.com/paper/2601.00121/) (2025)
- [Differentiable Discrete Event Simulation for Queuing Network Control](https://mlsanity.com/paper/2409.03740/) (2024)
- [Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2409.14557/) (2024)
- [Speeding up Policy Simulation in Supply Chain RL](https://mlsanity.com/paper/2406.01939/) (2024)
- [Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients](https://mlsanity.com/paper/2605.14297/) (2026)
- [Optimal Hidden-Target Learning for Online Inventory Optimization on General Convex Sets](https://mlsanity.com/paper/2606.14679/) (2026)
- [SkillChain-Gym: A Benchmark for Reskilling-Aware Production-Inventory Control under Disruptions](https://mlsanity.com/paper/2606.17266/) (2026)
- [Hard Constraints, Smooth Gradients: Learning Feasible Inventory Policies via Differentiable Projection](https://mlsanity.com/paper/2608.02343/) (2026)
- [Benign Nonconvex Landscape for Policy Optimization: Infinite-Horizon Discounted MDPs with General State and Action Spaces](https://mlsanity.com/paper/2609.21433/) (2026)
- [Proximal Residual Value Functions for Consistent Planning and Real-Time Execution](https://mlsanity.com/paper/2609.23242/) (2026)

## Abstract (arXiv)

We argue that inventory management presents unique opportunities for the reliable application of deep reinforcement learning (DRL). To enable this, we emphasize and test two complementary techniques. The first is Hindsight Differentiable Policy Optimization (HDPO), which uses pathwise gradients from offline counterfactual simulations to directly and efficiently optimize policy performance. Unlike standard policy gradient methods that rely on high-variance score-function estimators, HDPO computes gradients by differentiating through the known system dynamics. Via extensive benchmarking, we show that HDPO recovers near-optimal policies in settings with known or bounded optima, is more robust than variants of the REINFORCE algorithm, and significantly outperforms generalized newsvendor heuristics on problems using real time series data. Our second technique aligns neural policy architectures with the topology of the inventory network. We exploit Graph Neural Networks (GNNs) as a natural inductive bias for encoding supply chain structure, demonstrate that they can represent optimal and near-optimal policies in two theoretical settings, and empirically show that they reduce data requirements across six diverse inventory problems. A key obstacle to progress in this area is the lack of standardized benchmark problems. To address this gap, we open-source a suite of benchmark environments, along with our full codebase, to promote transparency and reproducibility. All resources are available at github.com/MatiasAlvo/Neural_inventory_control.