mlsanityData to 7 Oct 2026

Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model

arXiv 2412.13630 ↗First posted 18 Dec 2024cs.RO cs.AI cs.LGcited by 60

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Recent advancements in robot learning have used imitation learning with large models and extensive demonstrations to develop effective policies. However, these models are often limited by the quantity, quality, and diversity of demonstrations. This paper explores improving offline-trained imitation learning models through online interactions with the environment. We introduce Policy Decorator, which uses a model-agnostic residual policy to refine large imitation learning models during online interactions. By implementing controlled exploration strategies, Policy Decorator enables stable, sample-efficient online learning. Our evaluation spans eight tasks across two benchmarks-ManiSkill and Adroit-and involves two state-of-the-art imitation learning models (Behavior Transformer and Diffusion Policy). The results show Policy Decorator effectively improves the offline-trained policies and preserves the smooth motion of imitation learning models, avoiding the erratic behaviors of pure RL policies. See our project page (https://policydecorator.github.io) for videos.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 62 of its 78 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

62 of 78 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transport30 Jun
  2. 2021Implicit Behavioral Cloning1 Sep
  3. 2020Offline Learning from Demonstrations and Unlabeled Experience27 Nov
  4. 2020Error Bounds of Imitating Policies and Environments22 Oct
  5. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023Segment Anythingcited by 8,158
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  6. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  13. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  14. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  15. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  16. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  17. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  18. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  19. 2024Diffusion Policy Policy Optimizationcited by 255
  20. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  21. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  22. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  23. 2024Behavior Generation with Latent Actionscited by 159
  24. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  25. 2018Residual Policy Learningcited by 146
  26. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
  27. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  28. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  29. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  30. 2022Jump-Start Reinforcement Learningcited by 102
  31. 2018Playing hard exploration games by watching YouTubecited by 98
  32. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  33. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  34. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  35. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  36. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  37. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  38. 2015Learning to Search Better Than Your Teachercited by 50

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 40.

60 papers.

Sort:

By area:

  1. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  2. 2026WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RLcited by 46
  3. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  4. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  5. 2026Q-learning with Adjoint Matchingcited by 33
  6. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  7. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
  8. 2025Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Correctionscited by 25
  9. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  10. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  11. 2026VLS: Steering Pretrained Robot Policies via Vision-Language Modelscited by 14
  12. 2025Responsive Noise-Relaying Diffusion Policy: Responsive and Efficient Visuomotor Controlcited by 13
38 more
  1. 2025Touch begins where vision ends: Generalizable policies for contact-rich manipulationcited by 13
  2. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 11
  3. 2026RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulationcited by 10
  4. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  5. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 8
  6. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  7. 2025SENTINEL: A Fully End-to-End Language-Action Model for Humanoid Whole Body Controlcited by 7
  8. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 7
  9. 2025SIME: Enhancing Policy Self-Improvement with Modal-level Explorationcited by 6
  10. 2025Accelerating Residual Reinforcement Learning with Uncertainty Estimationcited by 6
  11. 2025IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuningcited by 5
  12. 2025SOE: Sample-Efficient Robot Policy Self-Improvement via On-Manifold Explorationcited by 5
  13. 2026Self-Correcting VLA: Online Action Refinement via Sparse World Imaginationcited by 4
  14. 2026TORL-VLA: Tactile Guided Online Reinforcement Learning for Contact-Rich Manipulationcited by 4
  15. 2025LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrationscited by 3
  16. 2026What Makes Value Learning Efficient in Residual Reinforcement Learning?cited by 3
  17. 2026When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learningcited by 3
  18. 2026Contrastive Conceptor Activation Steering (COAST): Unlocking Vision-Language-Action Models through Hidden Statescited by 3
  19. 2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
  20. 2026MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planningcited by 3
  21. 2025Factorizing Diffusion Policies for Observation Modality Prioritizationcited by 2
  22. 2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Controlcited by 2
  23. 2026FAR-Dex: Few-shot Data Augmentation and Adaptive Residual Policy Refinement for Dexterous Manipulationcited by 2
  24. 2026PPGuide: Steering Diffusion Policies with Performance Predictive Guidancecited by 2
  25. 2026Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policiescited by 2
  26. 2026DyGRO-VLA: Cross-Task Scaling of Vision-Language-Action Models via Dynamic Grouped Residual Optimizationcited by 2
  27. 2026ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policiescited by 2
  28. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 2
  29. 2025FoldNet: Learning Generalizable Closed-Loop Policy for Garment Folding via Keypoint-Driven Asset and Demonstration Synthesiscited by 1
  30. 2026ExpertGen: Scalable Sim-to-Real Expert Policy Learning from Imperfect Behavior Priorscited by 1
  31. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 1
  32. 2026Entropy-Regularized Adjoint Matching for Offline Reinforcement Learningcited by 1
  33. 2026Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latencycited by 1
  34. 2026Facet-0: A Robotic Foundation Model for Contact-Rich Precise Manipulationcited by 1
  35. 2025Fine-Tuning Flow Matching via Maximum Likelihood Estimation of Reconstructionscited by 0
  36. 2026Accelerating Robotic Reinforcement Learning with Agent Guidancecited by 0
  37. 2026PRISM: Personalized Refinement of Imitation Skills for Manipulation via Human Instructionscited by 0
  38. 2026Closed-Loop Neural Activation Control in Vision-Language-Action Modelscited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Imitation Learning with Conservative World Models
  2. 2025Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach
  3. 2020Explaining Fast Improvement in Online Imitation Learning
  4. 2019On-Policy Robot Imitation Learning from a Converging Supervisor
  5. 2024Online Adaptation for Enhancing Imitation Learning Policies
  6. 2025Dynamic Rank Adjustment in Diffusion Policies for Efficient and Flexible Training
  7. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  8. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning