mlsanityData to 14 Oct 2026

Imitate Optimal Policy: Prevail and Induce Action Collapse in Policy Gradient

arXiv 2509.02737 ↗First posted 2 Sep 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Policy gradient (PG) methods in reinforcement learning frequently utilize deep neural networks (DNNs) to learn a shared backbone of feature representations used to compute likelihoods in an action selection layer. Numerous studies have been conducted on the convergence and global optima of policy networks, but few have analyzed representational structures of those underlying networks. While training an optimal policy DNN, we observed that under certain constraints, a gentle structure resembling neural collapse, which we refer to as Action Collapse (AC), emerges. This suggests that 1) the state-action activations (i.e. last-layer features) sharing the same optimal actions collapse towards those optimal actions respective mean activations; 2) the variability of activations sharing the same optimal actions converges to zero; 3) the weights of action selection layer and the mean activations collapse to a simplex equiangular tight frame (ETF). Our early work showed those aforementioned constraints to be necessary for these observations. Since the collapsed ETF of optimal policy DNNs maximally separates the pair-wise angles of all actions in the state-action space, we naturally raise a question: can we learn an optimal policy using an ETF structure as a (fixed) target configuration in the action selection layer? Our analytical proof shows that learning activations with a fixed ETF as action selection layer naturally leads to the AC. We thus propose the Action Collapse Policy Gradient (ACPG) method, which accordingly affixes a synthetic ETF as our action selection layer. ACPG induces the policy DNN to produce such an ideal configuration in the action selection layer while remaining optimal. Our experiments across various OpenAI Gym environments demonstrate that our technique can be integrated into any discrete PG methods and lead to favorable reward improvements more quickly and robustly.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 33 of its 51 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

33 of 51 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  8. 2023Instruction Tuning with GPT-4cited by 686
  9. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  10. 2018Is Q-learning Provably Efficient?cited by 487
  11. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  12. 2018Understanding the impact of entropy on policy optimizationcited by 176
21 more
  1. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  2. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  3. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  4. 2020Neural collapse with unconstrained featurescited by 142
  5. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  6. 2020Phasic Policy Gradientcited by 123
  7. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 116
  8. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 106
  9. 2019Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPscited by 105
  10. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  11. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 52
  12. 2020Neural Collapse with Cross-Entropy Losscited by 52
  13. 2022Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learningcited by 43
  14. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 38
  15. 2021Regularized Policies are Reward Robustcited by 32
  16. 2023Inducing Neural Collapse in Deep Long-tailed Learningcited by 28
  17. 2023TD Convergence: An Optimization Perspectivecited by 16
  18. 2023Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severitycited by 13
  19. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  20. 2023On the Global Convergence of Natural Actor-Critic with Two-layer Neural Network Parametrizationcited by 5
  21. 2019Learning Classifiers on Positive and Unlabeled Data with Policy Gradientcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Learning Rates Go Wrong: Early Structural Signals in PPO Actor-Critic
  2. 2026Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States
  3. 2026Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithms
  4. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Space
  5. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO
  6. 2026Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching
  7. 2020Neural Dynamic Policies for End-to-End Sensorimotor Learning
  8. 2025Exploration Behavior of Untrained Policies