mlsanityData to 14 Oct 2026

Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidance

arXiv 2504.18766 ↗First posted 26 Apr 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning (RL) suffers from severe sample inefficiency, especially during early training, requiring extensive environmental interactions to perform competently. Existing methods tend to solve this by incorporating prior knowledge, but introduce significant architectural and implementation complexity. We propose Dynamic Action Interpolation (DAI), a universal yet straightforward framework that interpolates expert and RL actions via a time-varying weight α(t), integrating into any Actor-Critic algorithm with just a few lines of code and without auxiliary networks or additional losses. Our theoretical analysis shows that DAI reshapes state visitation distributions to accelerate value function learning while preserving convergence guarantees. Empirical evaluations across MuJoCo continuous control tasks demonstrate that DAI improves early-stage performance by over 160% on average and final performance by more than 50%, with the Humanoid task showing a 4× improvement early on and a 2× gain at convergence. These results challenge the assumption that complex architectural modifications are necessary for sample-efficient reinforcement learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 39 references that are in the map. First posted on arXiv 26 Apr 2025.

Builds on

34 of 39 references.

References · 34

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  9. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  12. 2018Residual Reinforcement Learning for Robot Controlcited by 274
22 more
  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  2. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  3. 2022Online Decision Transformercited by 196
  4. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  5. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  6. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  7. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  8. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 48
  9. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  10. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  11. 2020Behavior Priors for Efficient Reinforcement Learningcited by 41
  12. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
  13. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  14. 2021Residual Reinforcement Learning from Demonstrationscited by 31
  15. 2020Residual Learning from Demonstration: Adapting DMPs for Contact-rich Manipulationcited by 28
  16. 2018Mix&Match - Agent Curricula for Reinforcement Learningcited by 27
  17. 2022Minimax Optimal Online Imitation Learning via Replay Estimationcited by 25
  18. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  19. 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 17
  20. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Modelscited by 14
  21. 2023How To Guide Your Learner: Imitation Learning with Active Adaptive Expert Involvementcited by 8
  22. 2022A Joint Imitation-Reinforcement Learning Framework for Reduced Baseline Regretcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control
  2. 2025Distributions as Actions: A Unified Framework for Diverse Action Spaces
  3. 2026Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RL
  4. 2025SAC-MoE: Reinforcement Learning with Mixture-of-Experts for Control of Hybrid Dynamical Systems with Uncertainty
  5. 2026CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning
  6. 2026FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion
  7. 2026REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement
  8. 2026Dynamical Priors as a Training Objective in Reinforcement Learning