mlsanityData to 7 Oct 2026

Learning to Understand Goal Specifications by Modelling Reward

arXiv 1806.01946 ↗First posted 5 Jun 2018cs.AI cs.LGcited by 80

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations---and for instructions---not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 38 references that are in the map. First posted on arXiv 5 Jun 2018.

Builds on

20 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 19

  1. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  5. 2015Neural Module Networkscited by 375
  6. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  7. 2018Zero-Shot Visual Imitationcited by 114
  8. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  9. 2017Third-Person Imitation Learningcited by 103
  10. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  11. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  12. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
7 more
  1. 2017Gated-Attention Architectures for Task-Oriented Language Groundingcited by 72
  2. 2018Synthesizing Programs for Images using Reinforced Adversarial Learningcited by 72
  3. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  4. 2017Representation Learning for Grounded Spatial Reasoningcited by 23
  5. 2015Alignment-based compositional semantics for instruction followingcited by 21
  6. 2017HoME: a Household Multimodal Environmentcited by 20
  7. 2017Programmable Agentscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 2; 2020: 3; 2021: 18; 2022: 19; 2023: 17; 2024: 10; 2025: 6; 2026: 4.

80 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  4. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  5. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  6. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  7. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  8. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  9. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  10. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  11. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 72
  12. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
38 more
  1. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
  2. 2022Inferring Rewards from Language in Contextcited by 50
  3. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  4. 2021Adversarially Guided Actor-Criticcited by 47
  5. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 46
  6. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  7. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  8. 2020Learning Rewards from Linguistic Feedbackcited by 41
  9. 2019Positive-Unlabeled Reward Learningcited by 36
  10. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  11. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  12. 2021LILA: Language-Informed Latent Actionscited by 27
  13. 2022Leveraging Language for Accelerated Learning of Tool Manipulationcited by 26
  14. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  15. 2023Reward Collapse in Aligning Large Language Modelscited by 23
  16. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  17. 2020Inverse Reinforcement Learning with Natural Language Goalscited by 20
  18. 2022Language Control Diffusion: Efficiently Scaling through Space, Time, and Taskscited by 20
  19. 2023Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignmentcited by 19
  20. 2024Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learningcited by 18
  21. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  22. 2022Language and Culture Internalisation for Human-Like Autotelic AIcited by 14
  23. 2021Generalization in Text-based Games via Hierarchical Reinforcement Learningcited by 11
  24. 2022Perceiving the World: Question-guided Reinforcement Learning for Text-based Gamescited by 10
  25. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 10
  26. 2023Self-driven Grounding: Large Language Model Agents with Automatical Language-aligned Skill Learningcited by 9
  27. 2023Conceptual Reinforcement Learning for Language-Conditioned Taskscited by 8
  28. 2023Improving Long-Horizon Imitation Through Instruction Predictioncited by 8
  29. 2023Designing Fiduciary Artificial Intelligencecited by 8
  30. 2021Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translationcited by 7
  31. 2021Grounding Spatio-Temporal Language with Transformerscited by 7
  32. 2021SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agentscited by 7
  33. 2022Preprocessing Reward Functions for Interpretabilitycited by 7
  34. 2023Axiomatic Preference Modeling for Longform Question Answeringcited by 6
  35. 2021Interactive Hierarchical Guidance using Languagecited by 5
  36. 2022Safe Deep RL in 3D Environments using Human Feedbackcited by 5
  37. 2022Inverse Reinforcement Learning for Text Summarizationcited by 5
  38. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Datacited by 5

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
  2. 2026Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning
  3. 2023Reward Design with Language Models
  4. 2024Words as Beacons: Guiding RL Agents with High-Level Language Prompts
  5. 2025LinguaFluid: Language Guided Fluid Control via Semantic Rewards in Reinforcement Learning
  6. 2026Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
  7. 2026The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination
  8. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models