mlsanityData to 7 Oct 2026

Language-Critique Imitation Learning from Suboptimal Demonstrations

arXiv 2607.01225 ↗First posted 1 Jul 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Prior work on imitation learning from suboptimal demonstrations typically relies on compressed supervision signals such as confidence estimates, discriminator scores, or importance weights. These scalar signals are inherently limited, as they cannot explicitly express intermediate reasoning about task progress, failure modes, or corrective actions. We propose a language-critique framework for imitation learning from suboptimal demonstrations that instead leverages natural language as a structured supervision signal, avoiding the collapse of expressive feedback into scalars. Our method first constructs language labels from demonstrations that explicitly describe current progress, identify suboptimal behaviors, and provide fine-grained corrective guidance. We then introduce a language-critique loss that directly trains policies using these structured signals without reducing them to scalars, and instantiate it for both behavior cloning and diffusion policies, yielding LC-BC and LC-DP. We further provide a theoretical result showing that the proposed objective upper-bounds the expert performance gap under standard assumptions. Empirically, we evaluate on diverse continuous control tasks spanning navigation, manipulation, and gameplay, where our methods consistently outperform strong imitation learning and offline reinforcement learning baselines. These results demonstrate that language can serve as a powerful and structured form of supervision for learning robust policies from suboptimal data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 73 of its 99 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

73 of 99 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations16 May
  2. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning23 Sep
  3. 2020Language-Conditioned Imitation Learning for Robot Manipulation Tasks22 Oct
  4. 2020Language Conditioned Imitation Learning over Unstructured Data15 May

Other references · 69

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2023Mistral 7Bcited by 3,065
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  11. 2016Generative Adversarial Imitation Learningcited by 1,576
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
38 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  9. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  10. 2021Implicit Behavioral Cloningcited by 442
  11. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  12. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  13. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  14. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  15. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  16. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  17. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  18. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  19. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  20. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  21. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  22. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  23. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  24. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  25. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  26. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  27. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  28. 2019Imitation Learning from Imperfect Demonstrationcited by 112
  29. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  30. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  31. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  32. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  33. 2024Simple Hierarchical Planning with Diffusioncited by 71
  34. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  35. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  36. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  37. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  38. 2023Diffusion Model-Augmented Behavioral Cloningcited by 37

50 most cited of 69 All 69 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Tasks
  2. 2023Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations
  3. 2024Policy Improvement using Language Feedback Models
  4. 2024Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker
  5. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  6. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data
  7. 2024Learning with Language-Guided State Abstractions
  8. 2026Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning