mlsanityData to 14 Oct 2026

REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement

arXiv 2609.21108 ↗First posted 17 Sep 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Deep reinforcement learning (DRL) has achieved strong performance across a wide range of continuous-control problems. These continuous-control policies, however, are often defined as direct mappings from an observed state to an action or action distribution, requiring a single feed-forward network to construct an optimal control decision in one pass. While effective, this formulation leaves little opportunity for the policy to reconsider or progressively improve an action once an initial prediction has been formed. In this work, we explore an alternative approach: rather than learning only to directly predict an action, can a policy learn to iteratively improve one, and can this iterative process provide advantages during policy learning? We introduce Iterative Action Refinement (IAR), an iterative action-construction method that constructs control actions through a sequence of learned residual corrections. Starting from an initial proposal, a shared refinement network repeatedly conditions on the observed state and the current action proposal, allowing each refinement step to revise the action constructed by preceding steps. The final refined proposal is then used to determine the action executed by the agent. We integrate this iterative action-construction mechanism with Proximal Policy Optimization (PPO), yielding REFINEPPO. We evaluate REFINEPPO across 14 benchmark control tasks, complemented by controlled ablations of refinement depth and update schedules and analyses aimed at understanding why iterative refinement is effective. Across these environments, REFINEPPO matches or exceeds the performance of standard PPO while demonstrating faster convergence on several tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 29 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

23 of 29 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2018Neural Ordinary Differential Equationscited by 3,779
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  9. 2015Prioritized Experience Replaycited by 1,099
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  11. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  12. 2016Learning to learn by gradient descent by gradient descentcited by 620
11 more
  1. 2019Deep Equilibrium Modelscited by 615
  2. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  3. 2023Looped Transformers as Programmable Computerscited by 237
  4. 2018Residual Policy Learningcited by 164
  5. 2016Value Iteration Networkscited by 160
  6. 2018Iterative Amortized Inferencecited by 94
  7. 2024IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learningcited by 7
  8. 2024Generalizing Cooperative Eco-driving via Multi-residual Task Learningcited by 5
  9. 2026Prototype Guided Post-pretraining for Single-Cell Representation Learningcited by 3
  10. 2023Model-free Learning of Corridor Clearance: A Near-term Deployment Perspectivecited by 2
  11. 2025Multi-residual Mixture of Experts Learning for Cooperative Control in Multi-vehicle Systemscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States
  2. 2025Refined Policy Distillation: From VLA Generalists to RL Experts
  3. 2025Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidance
  4. 2026Learning When to Refine: Long-Horizon Reinforcement Learning for Budgeted Neural-Operator PDE Solvers
  5. 2025Enhancing Deep Deterministic Policy Gradients on Continuous Control Tasks with Decoupled Prioritized Experience Replay
  6. 2025LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation
  7. 2025Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions
  8. 2024Action Mapping for Reinforcement Learning in Continuous Environments with Constraints