# A Reinforcement Learning Method for Environments with Stochastic Variables: Post-Decision Proximal Policy Optimization with Dual Critic Networks

arXiv: https://arxiv.org/abs/2504.05150 · HTML: https://mlsanity.com/paper/2504.05150/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 10 of 10)

- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Noise-Aware Unsupervised Deep Lidar-Stereo Fusion](https://mlsanity.com/paper/1904.03868/) (2019)
- [OpenStereo: A Comprehensive Benchmark for Stereo Matching and Strong Baseline](https://mlsanity.com/paper/2312.00343/) (2023)
- [PVStereo: Pyramid Voting Module for End-to-End Self-Supervised Stereo Matching](https://mlsanity.com/paper/2103.07094/) (2021)
- [3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization](https://mlsanity.com/paper/1904.02917/) (2019)
- [Volumetric Propagation Network: Stereo-LiDAR Fusion for Long-Range Depth Estimation](https://mlsanity.com/paper/2103.12964/) (2021)
- [Embedded real-time stereo estimation via Semi-Global Matching on the GPU](https://mlsanity.com/paper/1610.04121/) (2016)
- [LiStereo: Generate Dense Depth Maps from LIDAR and Stereo Imagery](https://mlsanity.com/paper/1905.02744/) (2019)
- [Discontinuous and Smooth Depth Completion with Binary Anisotropic Diffusion Tensor](https://mlsanity.com/paper/2006.14374/) (2020)
- [Non-learning Stereo-aided Depth Completion under Mis-projection via Selective Stereo Matching](https://mlsanity.com/paper/2210.01436/) (2022)

## Abstract (arXiv)

This paper presents Post-Decision Proximal Policy Optimization (PDPPO), a novel variation of the leading deep reinforcement learning method, Proximal Policy Optimization (PPO). The PDPPO state transition process is divided into two steps: a deterministic step resulting in the post-decision state and a stochastic step leading to the next state. Our approach incorporates post-decision states and dual critics to reduce the problem's dimensionality and enhance the accuracy of value function estimation. Lot-sizing is a mixed integer programming problem for which we exemplify such dynamics. The objective of lot-sizing is to optimize production, delivery fulfillment, and inventory levels in uncertain demand and cost parameters. This paper evaluates the performance of PDPPO across various environments and configurations. Notably, PDPPO with a dual critic architecture achieves nearly double the maximum reward of vanilla PPO in specific scenarios, requiring fewer episode iterations and demonstrating faster and more consistent learning across different initializations. On average, PDPPO outperforms PPO in environments with a stochastic component in the state transition. These results support the benefits of using a post-decision state. Integrating this post-decision state in the value function approximation leads to more informed and efficient learning in high-dimensional and stochastic environments.