mlsanityData to 14 Oct 2026

Policy Improvement using Language Feedback Models

arXiv 2402.07876 ↗First posted 12 Feb 2024cs.LG cs.AI cs.CLcited by 16

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

We introduce Language Feedback Models (LFMs) that identify desirable behaviour - actions that help achieve tasks specified in the instruction - for imitation learning in instruction following. To train LFMs, we obtain feedback from Large Language Models (LLMs) on visual trajectories verbalized to language descriptions. First, by using LFMs to identify desirable behaviour to imitate, we improve in task-completion rate over strong behavioural cloning baselines on three distinct language grounding environments (Touchdown, ScienceWorld, and ALFWorld). Second, LFMs outperform using LLMs as experts to directly predict actions, when controlling for the number of LLM output tokens. Third, LFMs generalize to unseen environments, improving task-completion rate by 3.5-12.0% through one round of adaptation. Finally, LFM can be modified to provide human-interpretable feedback without performance loss, allowing human verification of desirable behaviour for imitation learning.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 51 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

30 of 51 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
18 more
  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  3. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  5. 2018Behavioral Cloning from Observationcited by 492
  6. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  7. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  8. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  9. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  10. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  11. 2017Unified Pragmatic Models for Generating and Following Instructionscited by 55
  12. 2019Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flightcited by 43
  13. 2019RTFM: Generalising to Novel Environment Dynamics via Readingcited by 33
  14. 2023Augmenting Autotelic Agents with Large Language Modelscited by 30
  15. 2021Language Models are Few-Shot Butlerscited by 27
  16. 2022Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areascited by 24
  17. 2015Alignment-based compositional semantics for instruction followingcited by 21
  18. 2021SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmarkcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
  2. 2025A Survey on Large Language Models for Automated Planningcited by 17
  3. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13
  4. 2025TALES: Text Adventure Learning Environment Suitecited by 11
  5. 2024Language Guided Exploration for RL Agents in Text Environmentscited by 9
  6. 2024Enhancing Agent Learning through World Dynamics Modelingcited by 9
  7. 2026PABU: Progress-Aware Belief Update for Efficient LLM Agentscited by 7
  8. 2026Learning from Language Feedback via Variational Policy Distillationcited by 7
  9. 2026AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agentscited by 7
  10. 2024Evaluating Vision-Language Models as Evaluators in Path Planningcited by 6
  11. 2024Natural Language Reinforcement Learningcited by 5
  12. 2025EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLMcited by 5
4 more
  1. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  2. 2025How well can LLMs provide planning feedback in grounded environments?cited by 1
  3. 2026A History-Aware Visually Grounded Critic for Computer Use Agentscited by 1
  4. 2025FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Training Language Models with Language Feedback at Scale
  2. 2026Language-Critique Imitation Learning from Suboptimal Demonstrations
  3. 2024Aligning Language Models with Demonstrated Feedback
  4. 2026Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention
  5. 2023Interpreting Learned Feedback Patterns in Large Language Models
  6. 2024Learning with Language-Guided State Abstractions
  7. 2020Language-Conditioned Imitation Learning for Robot Manipulation Tasks
  8. 2026When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models