mlsanityData to 14 Oct 2026

Selective Sampling and Imitation Learning via Online Regression

arXiv 2307.04998 ↗First posted 11 Jul 2023cs.LG cs.AI math.STcited by 12

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We consider the problem of Imitation Learning (IL) by actively querying noisy expert for feedback. While imitation learning has been empirically successful, much of prior work assumes access to noiseless expert feedback which is not practical in many applications. In fact, when one only has access to noisy expert feedback, algorithms that rely on purely offline data (non-interactive IL) can be shown to need a prohibitively large number of samples to be successful. In contrast, in this work, we provide an interactive algorithm for IL that uses selective sampling to actively query the noisy expert for feedback. Our contributions are twofold: First, we provide a new selective sampling algorithm that works with general function classes and multiple actions, and obtains the best-known bounds for the regret and the number of queries. Next, we extend this analysis to the problem of IL with noisy expert feedback and provide a new IL algorithm that makes limited queries. Our algorithm for selective sampling leverages function approximation, and relies on an online regression oracle w.r.t.~the given model class to predict actions, and to decide whether to query the expert for its label. On the theoretical side, the regret bound of our algorithm is upper bounded by the regret of the online regression oracle, while the query complexity additionally depends on the eluder dimension of the model class. We complement this with a lower bound that demonstrates that our results are tight. We extend our selective sampling algorithm for IL with general function approximation and provide bounds on both the regret and the number of queries made to the noisy expert. A key novelty here is that our regret and query complexity bounds only depend on the number of times the optimal policy (and not the noisy expert, or the learner) go to states that have a small margin.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 13 of its 49 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

13 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms14 Jun
  2. 2020Active Imitation Learning with Noisy Guidance26 May
  3. 2018Behavioral Cloning from Observation4 May

Other references · 10

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2016OpenAI Gymcited by 1,736
  3. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
  4. 2020Instance-Dependent Complexity of Contextual Bandits and Reinforcement Learning: A Disagreement-Based Perspectivecited by 78
  5. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  6. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 58
  7. 2022Efficient Active Learning with Abstentioncited by 22
  8. 2015Online Nonparametric Regression with General Loss Functionscited by 14
  9. 2020On the Sample Complexity of Reinforcement Learning with Policy Space Generalizationcited by 7
  10. 2022Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrationscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 4; 2026: 2.

12 papers.

  1. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  2. 2023When is Agnostic Reinforcement Learning Statistically Tractable?cited by 9
  3. 2025System-Aware Unlearning Algorithms: Use Lesser, Forget Fastercited by 4
  4. 2023Contextual Bandits and Imitation Learning via Preference-Based Active Queriescited by 2
  5. 2024Online Learning with Unknown Constraintscited by 2
  6. 2024Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloningcited by 2
  7. 2023Agnostic Interactive Imitation Learning: New Theory and Practical Algorithmscited by 1
  8. 2025Towards Fundamental Limits for Active Multi-distribution Learningcited by 1
  9. 2025Active Learning via Regression Beyond Realizabilitycited by 0
  10. 2025The Good, the Bad, and the Sampled: a No-Regret Approach to Safe Online Classificationcited by 0
  11. 2026Imitation Learning for Combinatorial Optimisation under Uncertaintycited by 0
  12. 2026When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Minimax Optimal Online Imitation Learning via Replay Estimation
  2. 2022On Efficient Online Imitation Learning via Classification
  3. 2025Sample-Efficient Expert Query Control in Active Imitation Learning via Conformal Prediction
  4. 2023A Strong Baseline for Batch Imitation Learning
  5. 2025A Model-Based Approach to Imitation Learning through Multi-Step Predictions
  6. 2026Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback
  7. 2020Explaining Fast Improvement in Online Imitation Learning
  8. 2022Improved Policy Optimization for Online Imitation Learning