mlsanityData to 7 Oct 2026

Imitation Bootstrapped Reinforcement Learning

arXiv 2311.02198 ↗First posted 3 Nov 2023cs.LG cs.AIcited by 54

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Despite the considerable potential of reinforcement learning (RL), robotic control tasks predominantly rely on imitation learning (IL) due to its better sample efficiency. However, it is costly to collect comprehensive expert demonstrations that enable IL to generalize to all possible scenarios, and any distribution shift would require recollecting data for finetuning. Therefore, RL is appealing if it can build upon IL as an efficient autonomous self-improvement procedure. We propose imitation bootstrapped reinforcement learning (IBRL), a novel framework for sample-efficient RL with demonstrations that first trains an IL policy on the provided demonstrations and then uses it to propose alternative actions for both online exploration and bootstrapping target values. Compared to prior works that oversample the demonstrations or regularize RL with an additional imitation loss, IBRL is able to utilize high quality actions from IL policies since the beginning of training, which greatly accelerates exploration and training efficiency. We evaluate IBRL on 6 simulation and 3 real-world tasks spanning various difficulty levels. IBRL significantly outperforms prior methods and the improvement is particularly more prominent in harder tasks.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 52 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

42 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Planning for Sample Efficient Imitation Learning18 Oct
  2. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transport30 Jun
  3. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Search14 Dec

Other references · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2018Group Normalizationcited by 1,514
  9. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2020Mastering Atari with Discrete World Modelscited by 990
  12. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
27 more
  1. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  3. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  4. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  5. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  6. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  7. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  8. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  9. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  10. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  11. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  12. 2022Masked World Models for Visual Controlcited by 169
  13. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  14. 2018Residual Policy Learningcited by 146
  15. 2020Visual Imitation Made Easycited by 111
  16. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  17. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  18. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  19. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  20. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61
  21. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  22. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  23. 2023HYDRA: Hybrid Robot Actions for Imitation Learningcited by 54
  24. 2022Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planningcited by 47
  25. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  26. 2020Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentationcited by 20
  27. 2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 21; 2026: 19.

54 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 255
  2. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  3. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  4. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  5. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  6. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  7. 2024RLDG: Robotic Generalist Policy Distillation via Reinforcement Learningcited by 42
  8. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  9. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  10. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
  11. 2025Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimizationcited by 20
  12. 2024So You Think You Can Scale Up Autonomous Robot Data Collection?cited by 17
38 more
  1. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  2. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15
  3. 2025Reinforcement Learning via Implicit Imitation Guidancecited by 12
  4. 2024Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learningcited by 10
  5. 2025Sketch-to-Skill: Bootstrapping Robot Learning with Human Drawn Trajectory Sketchescited by 9
  6. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 8
  7. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewardscited by 7
  8. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 7
  9. 2024PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learningcited by 6
  10. 2025Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transportcited by 6
  11. 2025TDMPBC: Self-Imitative Reinforcement Learning for Humanoid Robot Controlcited by 6
  12. 2025Accelerating Residual Reinforcement Learning with Uncertainty Estimationcited by 6
  13. 2024IMRL: Integrating Visual, Physical, Temporal, and Geometric Representations for Enhanced Food Acquisitioncited by 5
  14. 2025IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuningcited by 5
  15. 2026Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Drivingcited by 4
  16. 2026UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learningcited by 4
  17. 2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planningcited by 4
  18. 2025Think Outside the Policy: In-Context Steered Policy Optimizationcited by 3
  19. 2026When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learningcited by 3
  20. 2026Trust Region Q Adjoint Matchingcited by 3
  21. 2025Adaptive Conformal Guidance for Learning under Uncertaintycited by 2
  22. 2025Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulationcited by 2
  23. 2025A KL-regularization Framework for Learning to Plan with Adaptive Priorscited by 2
  24. 2026SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flowscited by 2
  25. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulationcited by 2
  26. 2026Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?cited by 2
  27. 2024Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Plannercited by 1
  28. 2024Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewardscited by 1
  29. 2025Solving Robotics Tasks with Prior Demonstration via Exploration-Efficient Deep Reinforcement Learningcited by 1
  30. 2025Iterative Refinement of Flow Policies in Probability Space for Online Reinforcement Learningcited by 1
  31. 2026Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learningcited by 1
  32. 2026XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policiescited by 1
  33. 2026From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Modelscited by 1
  34. 2026Adaptation of Generalist Robot Policies with Minimal Datacited by 1
  35. 2025Trajectory First: A Curriculum for Discovering Diverse Policiescited by 0
  36. 2025From Learning to Mastery: Achieving Safe and Efficient Real-World Autonomous Driving with Human-In-The-Loop Reinforcement Learningcited by 0
  37. 2026APC-RL: Exceeding Data-Driven Behavior Priors with Adaptive Policy Compositioncited by 0
  38. 2026Scalable Dexterous Robot Learning with AR-based Remote Human-Robot Interactionscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Divide & Conquer Imitation Learning
  2. 2024Accelerating Inverse Reinforcement Learning with Expert Bootstrapping
  3. 2023Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms
  4. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning
  5. 2023Bridging Imitation and Online Reinforcement Learning: An Optimistic Tale
  6. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges
  7. 2019A Divergence Minimization Perspective on Imitation Learning Methods
  8. 2023REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation