# Zero Collapse: A Failure Mode of Policy Gradient Methods in Discontinuous Reward Environments

arXiv: https://arxiv.org/abs/2605.30896 · HTML: https://mlsanity.com/paper/2605.30896/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 4 of its 9 references that are in the map.
- First posted on arXiv 29 May 2026.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 4 of 4)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)

## Abstract (arXiv)

Policy-gradient learning can deteriorate even after substantial initial improvement. We study this behavior in a repeated first-price auction setting with a sharp threshold between winning and losing bids. We identify a failure mode, zero collapse, in which a policy shifts into persistently near-zero reward as informative outcomes become rarely sampled. Experiments with REINFORCE and a TD state-value actor-critic illustrate this failure and evaluate practical mitigations. Adaptive update-magnitude control and smooth-output parameterization support sustained REINFORCE learning, while a learned baseline substantially stabilizes the fixed-rate comparison. Some actor-critic runs still collapse under these interventions. A diagnostic near one observed collapse reveals severe continuation-value errors that reverse the relative desirability of actions across the winning threshold. These results illustrate how thresholded outcomes, policy updates, and inaccurate value estimates can contribute to zero collapse, and demonstrate practical mitigations in the studied setting.