mlsanityData to 7 Oct 2026

Quantile Filtered Imitation Learning

arXiv 2112.00950 ↗First posted 2 Dec 2021cs.LG stat.MLcited by 6

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We introduce quantile filtered imitation learning (QFIL), a novel policy improvement operator designed for offline reinforcement learning. QFIL performs policy improvement by running imitation learning on a filtered version of the offline dataset. The filtering process removes s,a pairs whose estimated Q values fall below a given quantile of the pushforward distribution over values induced by sampling actions from the behavior policy. The definitions of both the pushforward Q distribution and resulting value function quantile are key contributions of our method. We prove that QFIL gives us a safe policy improvement step with function approximation and that the choice of quantile provides a natural hyperparameter to trade off bias and variance of the improvement step. Empirically, we perform a synthetic experiment illustrating how QFIL effectively makes a bias-variance tradeoff and we see that QFIL performs well on the D4RL benchmark.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 25 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

22 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning27 Oct

Other references · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2017Constrained Policy Optimizationcited by 928
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
9 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  3. 2020Critic Regularized Regressioncited by 269
  4. 2018What is the Effect of Importance Weighting in Deep Learning?cited by 250
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  6. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  7. 2021Offline RL Without Off-Policy Evaluationcited by 150
  8. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  9. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 0; 2025: 2.

6 papers.

  1. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  2. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  3. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learningcited by 10
  4. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  5. 2025Preference learning made easy: Everything should be understood through win ratecited by 3
  6. 2025Decision SpikeFormer: Spike-Driven Transformer for Decision Makingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning
  2. 2023Residual Q-Learning: Offline and Online Policy Customization without Value
  3. 2025Inverse Q-Learning Done Right: Offline Imitation Learning in Qπ-Realizable MDPs
  4. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching
  5. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning
  6. 2026From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning
  7. 2023Coherent Soft Imitation Learning
  8. 2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning