# Inferring Rewards from Language in Context

arXiv: https://arxiv.org/abs/2204.02515 · HTML: https://mlsanity.com/paper/2204.02515/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 5 Apr 2022.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Learning to Understand Goal Specifications by Modelling Reward](https://mlsanity.com/paper/1806.01946/) (2018)
- [From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following](https://mlsanity.com/paper/1902.07742/) (2019)
- [Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding](https://mlsanity.com/paper/1703.10186/) (2017)
- [Unified Pragmatic Models for Generating and Following Instructions](https://mlsanity.com/paper/1711.04987/) (2017)
- [Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences](https://mlsanity.com/paper/1506.04089/) (2015)
- [Learning Language Games through Interaction](https://mlsanity.com/paper/1606.02447/) (2016)
- [From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood](https://mlsanity.com/paper/1704.07926/) (2017)
- [Learning to Map Context-Dependent Sentences to Executable Formal Queries](https://mlsanity.com/paper/1804.06868/) (2018)
- [Alignment-based compositional semantics for instruction following](https://mlsanity.com/paper/1508.06491/) (2015)
- [Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularities](https://mlsanity.com/paper/1704.06616/) (2017)
- [Translating Neuralese](https://mlsanity.com/paper/1704.06960/) (2017)
- [Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation](https://mlsanity.com/paper/1805.10209/) (2018)
- [Extending rational models of communication from beliefs to actions](https://mlsanity.com/paper/2105.11950/) (2021)

## Built on by (most cited first, 30 of 50)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving](https://mlsanity.com/paper/2310.03026/) (2023)
- [Grounding Language Models to Images for Multimodal Inputs and Outputs](https://mlsanity.com/paper/2301.13823/) (2023)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [Building Machines that Learn and Think with People](https://mlsanity.com/paper/2408.03943/) (2024)
- [Inverse Preference Learning: Preference-based RL without a Reward Function](https://mlsanity.com/paper/2305.15363/) (2023)
- [Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft](https://mlsanity.com/paper/2312.09238/) (2023)
- [Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs](https://mlsanity.com/paper/2403.05020/) (2024)
- [Training Language Models with Language Feedback](https://mlsanity.com/paper/2204.14146/) (2022)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models](https://mlsanity.com/paper/2503.17523/) (2025)
- [Bayesian Preference Elicitation with Language Models](https://mlsanity.com/paper/2403.05534/) (2024)
- [Distilling Internet-Scale Vision-Language Models into Embodied Agents](https://mlsanity.com/paper/2301.12507/) (2023)
- [Decision-Oriented Dialogue for Human-AI Collaboration](https://mlsanity.com/paper/2305.20076/) (2023)
- [Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning](https://mlsanity.com/paper/2402.17930/) (2024)
- [Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans](https://mlsanity.com/paper/2209.13020/) (2022)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Learning Personalized Agents from Human Feedback](https://mlsanity.com/paper/2602.16173/) (2026)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [Preference-Conditioned Language-Guided Abstraction](https://mlsanity.com/paper/2402.03081/) (2024)
- [Value of Information: A Framework for Human-Agent Communication](https://mlsanity.com/paper/2601.06407/) (2026)
- [Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspective](https://mlsanity.com/paper/2402.00262/) (2024)
- [Problem Solving Through Human-AI Preference-Based Cooperation](https://mlsanity.com/paper/2408.07461/) (2024)
- [Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competencies](https://mlsanity.com/paper/2402.03628/) (2024)
- [Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learning](https://mlsanity.com/paper/2412.05515/) (2024)
- [Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards](https://mlsanity.com/paper/2301.10095/) (2023)
- [Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input](https://mlsanity.com/paper/2405.14769/) (2024)

## Abstract (arXiv)

In classic instruction following, language like "I'd like the JetBlue flight" maps to actions (e.g., selecting that flight). However, language also conveys information about a user's underlying reward function (e.g., a general preference for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flight-booking task with natural language, our model more accurately infers rewards and predicts optimal actions in unseen environments, in comparison to past work that first maps language to actions (instruction following) and then maps actions to rewards (inverse reinforcement learning).