# Computing optimal policies for managing inventories with noisy observations

arXiv: https://arxiv.org/abs/2507.19765 · HTML: https://mlsanity.com/paper/2507.19765/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 5 of its 24 references that are in the map.
- First posted on arXiv 26 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 5 of 5)

- [Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities](https://mlsanity.com/paper/1401.2168/) (2014)
- [Markov Decision Processes with Incomplete Information and Semi-Uniform Feller Transition Probabilities](https://mlsanity.com/paper/2108.09232/) (2021)
- [On the Convergence of Optimal Actions for Markov Decision Processes and the Optimality of (s,S) Inventory Policies](https://mlsanity.com/paper/1507.05125/) (2015)
- [Structure of Optimal Solutions to Periodic-Review Total-Cost Inventory Control Models with Convex Costs and Backorders for all Values of Discount Factors](https://mlsanity.com/paper/1609.03984/) (2016)
- [Continuity of Filters for Discrete-Time Control Problems Defined by Explicit Equations](https://mlsanity.com/paper/2311.12184/) (2023)

## Abstract (arXiv)

This paper implements the Deep Deterministic Policy Gradient (DDPG) algorithm for computing optimal policies for partially observable single-product periodic review inventory control problems with setup costs and backorders. The decision maker does not know the exact inventory level, but can obtain noise-corrupted observations of them. The goal is to maximize the expected total discounted costs incurred over a finite planning horizon. We also investigate the Gaussian version of this problem with normally distributed initial inventories, demands, and observation noise. We show that expected posterior observations of inventory levels, also called mean beliefs, provide sufficient statistics for the Gaussian problem. Moreover, they can be represented in the form of a Markov Decision Processes for an inventory control system with time-dependent holding costs and demands. Thus, for a Gaussian problem, the there exist (s_t,S_t)-optimal policies based on mean beliefs, and this fact explains the structure of the approximately optimal policies computed by DDPG. For the Gaussian case, we also numerically compare the performance of policies derived from DDPG to optimal policies for discretized versions of the original continuous problem.