mlsanityData to 7 Oct 2026

When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon

arXiv 2606.30445 ↗First posted 29 Jun 2026cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Online imitation learning (IL), particularly on-policy distillation, has emerged as a strong LLM post-training approach, often outperforming offline supervised fine-tuning (SFT). Yet a principled understanding of when and why online interaction helps remains unclear. In this work, we challenge the view that error accumulation is the main source of online IL's advantage, and instead show that the benefits of online interaction depend critically on whether the setting is realizable, i.e., whether the student policy class can represent the expert policy. Under realizability, we empirically find that offline IL already matches expert performance. In contrast, in non-realizable (misspecified) settings, we prove that offline IL encounters an information-theoretic bottleneck even when horizon H=1, and propose a structural characterization of misspecification relative to the reward, under which online IL provably achieves high performance despite a large distributional mismatch between the expert and student policies.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 68 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

51 of 68 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Efficient Imitation under Misspecification17 Mar
  2. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning20 Jul
  3. 2022Minimax Optimal Online Imitation Learning via Replay Estimation30 May
  4. 2020Toward the Fundamental Limits of Imitation Learning13 Sep

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
35 more
  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2016Sequence-Level Knowledge Distillationcited by 761
  5. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  6. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  7. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  8. 2015Policy Distillationcited by 414
  9. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  10. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  11. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  12. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  13. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  14. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  15. 2020Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oraclescited by 195
  16. 2026MiMo-V2-Flash Technical Reportcited by 163
  17. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  18. 2025Skywork Open Reasoner 1 Technical Reportcited by 161
  19. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  20. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  21. 2020Bypassing the Monster: A Faster and Simpler Optimal Algorithm for Contextual Bandits under Realizabilitycited by 114
  22. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  23. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  24. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  25. 2018Practical Contextual Bandits with Regression Oraclescited by 92
  26. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  27. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  28. 2025Black-Box On-Policy Distillation of Large Language Modelscited by 53
  29. 2025Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Explorationcited by 34
  30. 2023Foundations of Reinforcement Learning and Interactive Decision Makingcited by 33
  31. 2023Importance-Weighted Offline Learning Done Rightcited by 15
  32. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 14
  33. 2025Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecificationcited by 12
  34. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
  35. 2025To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Distillation as Probability Transport: Routed On-Policy Distillationcited by 0
  2. 2026Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Explaining Fast Improvement in Online Imitation Learning
  2. 2026Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback
  3. 2024Offline Imitation Learning by Controlling the Effective Planning Horizon
  4. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
  5. 2025Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learning
  6. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shift
  7. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
  8. 2025Minimal Ingredients for Reward Assignment from Expert Demonstrations