mlsanityData to 7 Oct 2026

Reward Design with Language Models

arXiv 2303.00001 ↗First posted 27 Feb 2023cs.LG cs.AI cs.CLcited by 257

Not on the map. Why?

Cited by 44 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require many expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by prompting a large language model (LLM) such as GPT-3 as a proxy reward function, where the user provides a textual prompt containing a few examples (few-shot) or a description (zero-shot) of the desired behavior. Our approach leverages this proxy reward function in an RL framework. Specifically, users specify a prompt once at the beginning of training. During training, the LLM evaluates an RL agent's behavior against the desired behavior described by the prompt and outputs a corresponding reward signal. The RL agent then uses this reward to update its behavior. We evaluate whether our approach can train agents aligned with user objectives in the Ultimatum Game, matrix games, and the DealOrNoDeal negotiation task. In all three tasks, we show that RL agents trained with our framework are well-aligned with the user's objectives and outperform RL agents trained with reward functions learned via supervised learning

Key facts. Cited by 257 papers in the map (data through 7 Oct 2026). Builds on 25 of its 40 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

25 of 40 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
13 more
  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  2. 2022Can language models learn from explanations in context?cited by 248
  3. 2017Inverse Reward Designcited by 246
  4. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  5. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 156
  6. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  7. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  8. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  9. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  10. 2022Inferring Rewards from Language in Contextcited by 50
  11. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  12. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  13. 2021Targeted Data Acquisition for Evolving Negotiation Agentscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 45; 2024: 96; 2025: 78; 2026: 38.

257 papers.

Sort:

By area:

  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  4. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  5. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  6. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  7. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  8. 2023Large Language Models as General Pattern Machinescited by 196
  9. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  10. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  11. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  12. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
38 more
  1. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  2. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  3. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  4. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  5. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  6. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  7. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  8. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  9. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  10. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  11. 2023Vision-Language Models as a Source of Rewardscited by 65
  12. 2023Strategic Reasoning with Language Modelscited by 64
  13. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61
  14. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 60
  15. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  16. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 59
  17. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  18. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 51
  19. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  20. 2023SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledgecited by 49
  21. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  22. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  23. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 44
  24. 2024Evaluating Language Model Agency through Negotiationscited by 43
  25. 2023ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifierscited by 41
  26. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
  27. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Spacecited by 40
  28. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 38
  29. 2023Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Roboticscited by 35
  30. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  31. 2025AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via GRPOcited by 33
  32. 2024Reward-Robust RLHF in LLMscited by 32
  33. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  34. 2023Large Language Model as a Policy Teacher for Training Reinforcement Learning Agentscited by 30
  35. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  36. 2025Safety Guardrails for LLM-Enabled Robotscited by 30
  37. 2024Controlling Large Language Model Agents with Entropic Activation Steeringcited by 29
  38. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29

50 most cited of 257 All 257 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
  2. 2026The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination
  3. 2026RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  5. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  6. 2025PARL: Prompt-based Agents for Reinforcement Learning
  7. 2024RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution
  8. 2018Learning to Understand Goal Specifications by Modelling Reward