mlsanityData to 7 Oct 2026

A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets

arXiv 2110.04698 ↗First posted 10 Oct 2021cs.LGcited by 5

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recent Offline Reinforcement Learning methods have succeeded in learning high-performance policies from fixed datasets of experience. A particularly effective approach learns to first identify and then mimic optimal decision-making strategies. Our work evaluates this method's ability to scale to vast datasets consisting almost entirely of sub-optimal noise. A thorough investigation on a custom benchmark helps identify several key challenges involved in learning from high-noise datasets. We re-purpose prioritized experience sampling to locate expert-level demonstrations among millions of low-performance samples. This modification enables offline agents to learn state-of-the-art policies in benchmark tasks using datasets where expert actions are outnumbered nearly 65:1.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 46 references that are in the map. First posted on arXiv 10 Oct 2021.

Builds on

36 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  2. 2020Critic Regularized Regression26 Jun
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 30

  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2017Deep Reinforcement Learning that Matterscited by 727
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2018An Algorithmic Perspective on Imitation Learningcited by 455
18 more
  1. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  2. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  4. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  5. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  6. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  7. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  8. 2018Self-Imitation Learningcited by 135
  9. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  10. 2019Imitation Learning from Imperfect Demonstrationcited by 112
  11. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  12. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  13. 2016Learning values across many orders of magnitudecited by 80
  14. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  15. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  16. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 64
  17. 2018Learning Self-Imitating Diverse Policiescited by 29
  18. 2020Self-Imitation Advantage Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 1; 2026: 1.

5 papers.

  1. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  2. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learningcited by 22
  3. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  4. 2025Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrationscited by 2
  5. 2026Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning via Implicit Imitation Guidance
  2. 2025Expert or not? assessing data quality in offline reinforcement learning
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  4. 2021A Dataset Perspective on Offline Reinforcement Learning
  5. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning
  6. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?
  7. 2025Offline Behavioral Data Selection
  8. 2026Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback