mlsanityData to 7 Oct 2026

Learning Rewards from Linguistic Feedback

arXiv 2009.14715 ↗First posted 30 Sep 2020cs.AIcited by 41

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We explore unconstrained natural language feedback as a learning signal for artificial agents. Humans use rich and varied language to teach, yet most prior work on interactive learning from language assumes a particular form of input (e.g., commands). We propose a general framework which does not make this assumption, using aspect-based sentiment analysis to decompose feedback into sentiment about the features of a Markov decision process. We then perform an analogue of inverse reinforcement learning, regressing the sentiment on the features to infer the teacher's latent reward function. To evaluate our approach, we first collect a corpus of teaching behavior in a cooperative task where both teacher and learner are human. We implement three artificial learners: sentiment-based "literal" and "pragmatic" models, and an inference network trained end-to-end to predict latent rewards. We then repeat our initial experiment and pair them with human teachers. All three successfully learn from interactive human feedback. The sentiment models outperform the inference network, with the "pragmatic" model approaching human performance. Our work thus provides insight into the information structure of naturalistic linguistic feedback as well as methods to leverage it for reinforcement learning.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 77 references that are in the map. First posted on arXiv 30 Sep 2020.

Builds on

24 of 77 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Learning to Understand Goal Specifications by Modelling Reward5 Jun
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 22

  1. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  2. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  3. 2016Cooperative Inverse Reinforcement Learningcited by 298
  4. 2019Vision-and-Dialog Navigationcited by 246
  5. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  6. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  7. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  8. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  9. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  10. 2019Executing Instructions in Situated Collaborative Interactionscited by 68
  11. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  12. 2017Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddingscited by 59
10 more
  1. 2018Training Classifiers with Natural Language Explanationscited by 58
  2. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  3. 2016Learning Language Games through Interactioncited by 43
  4. 2019A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Contextcited by 35
  5. 2020PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewardscited by 30
  6. 2017Beating Atari with Natural Language Guided Reinforcement Learningcited by 24
  7. 2020Inverse Reinforcement Learning with Natural Language Goalscited by 20
  8. 2019MeetUp! A Corpus of Joint Activity Dialogues in a Visual Environmentcited by 19
  9. 2019Why Build an Assistant in Minecraft?cited by 14
  10. 2019HIGhER : Improving instruction following with Hindsight Generation for Experience Replaycited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 8; 2023: 12; 2024: 11; 2025: 4; 2026: 2.

41 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023Cognitive Architectures for Language Agentscited by 422
  3. 2023Training Language Models with Language Feedback at Scalecited by 125
  4. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  5. 2024Building Machines that Learn and Think with Peoplecited by 73
  6. 2023Learning to Generate Better Than Your LLMcited by 49
  7. 2022Training Language Models with Language Feedbackcited by 48
  8. 2021Communicating Natural Programs to Humans and Machinescited by 35
  9. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  10. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  11. 2021Interactive Learning from Activity Descriptioncited by 26
  12. 2022Leveraging Language for Accelerated Learning of Tool Manipulationcited by 26
29 more
  1. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  2. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  3. 2024Natural Language Reinforcement Learningcited by 17
  4. 2021A Framework for Learning to Request Rich and Contextually Useful Information from Humanscited by 12
  5. 2024Learning with Language-Guided State Abstractionscited by 11
  6. 2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
  7. 2024Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competenciescited by 9
  8. 2023Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standardscited by 8
  9. 2022Linguistic communication as (inverse) reward designcited by 6
  10. 2024Retrospective Learning from Interactionscited by 6
  11. 2022AugCSE: Contrastive Sentence Embedding with Diverse Augmentationscited by 5
  12. 2023Curricular Subgoals for Inverse Reinforcement Learningcited by 5
  13. 2022RLang: A Declarative Language for Describing Partial World Knowledge to Reinforcement Learning Agentscited by 4
  14. 2024Natural Language Reinforcement Learningcited by 4
  15. 2023Simple Embodied Language Learning as a Byproduct of Meta-Reinforcement Learningcited by 3
  16. 2024Representational Alignment Supports Effective Machine Teachingcited by 3
  17. 2024Social Learning through Interactions with Other Agents: A Surveycited by 3
  18. 2023Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learningcited by 2
  19. 2025Increasing happiness through conversations with artificial intelligencecited by 2
  20. 2025Playpen: An Environment for Exploring Learning Through Conversational Interactioncited by 2
  21. 2023Improve the efficiency of deep reinforcement learning through semantic exploration guided by natural languagecited by 1
  22. 2023Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testingcited by 1
  23. 2024Trajectory-Oriented Policy Optimization with Sparse Rewardscited by 1
  24. 2025LLM-Driven Self-Refinement for Embodied Drone Task Planningcited by 1
  25. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignmentcited by 1
  26. 2021Neural Abstructions: Abstractions that Support Construction for Grounded Language Learningcited by 0
  27. 2022Teaching Drones on the Fly: Can Emotional Feedback Serve as Learning Signal for Training Artificial Agents?cited by 0
  28. 2025QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agentscited by 0
  29. 2026MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2026Reward Modeling from Natural Language Human Feedback
  4. 2023Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning
  5. 2019Fine-Tuning Language Models from Human Preferences
  6. 2024Improving Reward Models with Synthetic Critiques
  7. 2025Text2Grad: Reinforcement Learning from Natural Language Feedback
  8. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation