# Constrained Reinforcement Learning for the Dynamic Inventory Routing Problem under Stochastic Supply and Demand

arXiv: https://arxiv.org/abs/2503.05276 · HTML: https://mlsanity.com/paper/2503.05276/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 89 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Reinforcement Learning with Convex Constraints](https://mlsanity.com/paper/1906.09323/) (2019)
- [Reachability Constrained Reinforcement Learning](https://mlsanity.com/paper/2205.07536/) (2022)
- [Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2011.09999/) (2020)
- [Safety-Constrained Reinforcement Learning for MDPs](https://mlsanity.com/paper/1510.05880/) (2015)
- [Deep Q-Learning for Same-Day Delivery with Vehicles and Drones](https://mlsanity.com/paper/1910.11901/) (2019)
- [Deep Controlled Learning for Inventory Control](https://mlsanity.com/paper/2011.15122/) (2020)
- [Scenario Predict-then-Optimize for Data-Driven Online Inventory Routing](https://mlsanity.com/paper/2401.17787/) (2024)
- [Deep Policy Iteration with Integer Programming for Inventory Management](https://mlsanity.com/paper/2112.02215/) (2021)
- [Combinatorial Optimization and Machine Learning for Dynamic Inventory Routing](https://mlsanity.com/paper/2402.04463/) (2024)
- [Stochastic Cyclic Inventory Routing with Supply Uncertainty: A Case in Green-Hydrogen Logistics](https://mlsanity.com/paper/2212.10532/) (2022)

## Abstract (arXiv)

Green hydrogen has multiple use cases and is produced from renewable energy, such as solar or wind energy. It can be stored in large quantities, decoupling renewable energy generation from its use, and is therefore considered essential for achieving a climate-neutral economy. The intermittency of renewable energy generation and the stochastic nature of demand are, however, challenging factors for the dynamic planning of hydrogen storage and transportation. This holds particularly in the early-adoption phase when hydrogen distribution occurs through vehicle-based networks. We therefore address the Dynamic Inventory Routing Problem (DIRP) under stochastic supply and demand with direct deliveries for the vehicle-based distribution of hydrogen. To solve this problem, we propose a Constrained Reinforcement Learning (CRL) framework that integrates constraints into the learning process and incorporates parameterized post-decision state value predictions. Additionally, we introduce Lookahead-based CRL (LCRL), which improves decision-making over a multi-period horizon to enhance short-term planning while maintaining the value predictions. Our computational experiments demonstrate the efficacy of CRL and LCRL across diverse instances. Our learning methods provide near-optimal solutions on small scale instances that are solved via value iteration. Furthermore, both methods outperform typical deep learning approaches such as Proximal Policy Optimization, as well as classical inventory heuristics, such as (s,S)-policy-based and Power-of-Two-based heuristics. Furthermore, LCRL achieves a 10% improvement over CRL on average, albeit with higher computational requirements. Analyses of optimal replenishment policies reveal that accounting for stochastic supply and demand influences these policies, showing the importance of our addition to the DIRP.