mlsanityData to 14 Oct 2026

Action Shaping: Policies Absorb What They Can Express

arXiv 2609.32752 ↗First posted 26 Sep 2026cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Reward shaping has a theorem: a potential-based term can be removed without changing the optimal policy. The same practice on the action channel, an offset added in training and dropped at deployment, has no theorem. Nothing cancels an action offset, so the correction is kept at deployment or removed without a guarantee. We call it action shaping and state its principle. A trainable policy absorbs an offset its own output layer can reproduce exactly, which is what we mean by express; what is absorbed can be removed with the return intact. Its minimal instance is a zero-initialized linear head behind a learnable gate, added to an actor that trains through a learned action-value function, with no penalty or schedule. The gate rises and then falls on its own, for deterministic and stochastic actors alike, and on 20 tasks removing the head costs almost nothing. The condition is exact reproduction, not capacity: a nonlinear head with more parameters is not absorbed, and in a paired control, one linear path added to a nonlinear base head restores absorption. Exact reproduction gives the loss a flat direction that gradient noise drifts along, and the offset's amplitude indicates, before removal, what dropping the head will cost. Action shaping thus gains the counterpart of the shaping theorem, a condition for absorption, together with the mechanism behind it and a diagnostic that reads it. Policies absorb what they can express, and only that.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 56 of its 63 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

56 of 63 references.

References · 56

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018DeepMind Control Suitecited by 956
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2021Going deeper with Image Transformerscited by 764
  7. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 522
  8. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  9. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  10. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  11. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  12. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
38 more
  1. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  2. 2018Residual Policy Learningcited by 164
  3. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  4. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  5. 2022Jump-Start Reinforcement Learningcited by 112
  6. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  7. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  8. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  9. 2026SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalizationcited by 94
  10. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 78
  11. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 55
  12. 2018ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networkscited by 53
  13. 2020Action Space Shaping in Deep Reinforcement Learningcited by 46
  14. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  15. 2024Privileged Sensing Scaffolds Reinforcement Learningcited by 19
  16. 2022Revisiting Discrete Soft Actor-Criticcited by 13
  17. 2023DecAP: Decaying Action Priors for Accelerated Imitation Learning of Torque-Based Legged Locomotion Policiescited by 7
  18. 2024RaceMOP: Mapless Online Path Planning for Multi-Agent Autonomous Racing using Residual Policy Learningcited by 6
  19. 2026SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Trainingcited by 6
  20. 2026Harness-Zero: Harness Distillation via Agent-as-Harnesscited by 5
  21. 2026Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformerscited by 4
  22. 2026What Makes Value Learning Efficient in Residual Reinforcement Learning?cited by 4
  23. 2018Constrained Exploration and Recovery from Experience Shapingcited by 3
  24. 2026R2PO: Decoupling Rollout and Inference Policies for LLM Reasoningcited by 3
  25. 2024Efficient Multi-Task Reinforcement Learning via Task-Specific Action Correctioncited by 2
  26. 2025Residual Policy Gradient: A Reward View of KL-regularized Objectivecited by 2
  27. 2025APEX: Action Priors Enable Efficient Exploration for Robust Motion Tracking on Legged Robotscited by 2
  28. 2025Data-Free Pruning of Self-Attention Layers in LLMscited by 2
  29. 2026Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learningcited by 2
  30. 2026Exact Network Surgery: Functional Invariance and Gradient Plasticity in Reactive Computational Graphscited by 2
  31. 2026PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learningcited by 2
  32. 2026One Policy, Any Budget: Internalizing Budget-Aware Search via Reinforcement Learningcited by 2
  33. 2026Efficient Real-World Autonomous Racing via Attenuated Residual Policy Optimizationcited by 1
  34. 2026An Agency-Transferring Model-Free Policy Enhancement Techniquecited by 1
  35. 2026Learning through Internalizationcited by 1
  36. 2026MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basinscited by 1
  37. 2026DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Trainingcited by 1
  38. 2026Warp RL: Reshaping Base Policy Distributions for Dynamics Adaptationcited by 1

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.