mlsanityData to 7 Oct 2026

Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning

arXiv 2312.04736 ↗First posted 7 Dec 2023cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Despite numerous successes, the field of reinforcement learning (RL) remains far from matching the impressive generalisation power of human behaviour learning. One possible way to help bridge this gap be to provide RL agents with richer, more human-like feedback expressed in natural language. To investigate this idea, we first extend BabyAI to automatically generate language feedback from the environment dynamics and goal condition success. Then, we modify the Decision Transformer architecture to take advantage of this additional signal. We find that training with language feedback either in place of or in addition to the return-to-go or goal descriptions improves agents' generalisation performance, and that agents can benefit from feedback even when this is only available during training, but not at inference.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

45 of 58 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2018World Modelscited by 1,444
  10. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  11. 2023Mastering Diverse Domains through World Modelscited by 1,193
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
33 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  3. 2020Mastering Atari with Discrete World Modelscited by 990
  4. 2023Teaching Large Language Models to Self-Debugcited by 963
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  9. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  10. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  11. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  12. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  13. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  14. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  15. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  16. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  17. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  18. 2023Training Language Models with Language Feedback at Scalecited by 125
  19. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  20. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  21. 2023Shepherd: A Critic for Language Model Generationcited by 96
  22. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  23. 2021On the Expressivity of Markov Rewardcited by 71
  24. 2023Learning to Model the World with Languagecited by 66
  25. 2023Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoningcited by 49
  26. 2022Training Language Models with Language Feedbackcited by 48
  27. 2023Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Promptingcited by 45
  28. 2023LeTI: Learning to Generate from Textual Interactionscited by 25
  29. 2021ReaSCAN: Compositional Reasoning in Language Groundingcited by 19
  30. 2020Think before you act: A simple baseline for compositional generalizationcited by 11
  31. 2022When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarkscited by 11
  32. 2020CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learningcited by 8
  33. 2020Zero-Shot Compositional Policy Learning via Language Groundingcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024VeriGraph: Scene Graphs for Execution Verifiable Robot Planningcited by 7
  2. 2025FOSSIL: Harnessing Feedback on Suboptimal Samples for Data-Efficient Generalisation with Imitation Learning for Embodied Vision-and-Language Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
  2. 2020Learning Rewards from Linguistic Feedback
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learning
  4. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs
  5. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
  6. 2024RLVF: Learning from Verbal Feedback without Overgeneralization
  7. 2024Natural Language Reinforcement Learning
  8. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework