mlsanityData to 14 Oct 2026

Text2Reward: Reward Shaping with Language Models for Reinforcement Learning

arXiv 2309.11489 ↗First posted 20 Sep 2023cs.LG cs.AI cs.CLcited by 160

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Designing reward functions is a longstanding challenge in reinforcement learning (RL); it requires specialized knowledge or domain data, leading to high costs for development. To address this, we introduce Text2Reward, a data-free framework that automates the generation and shaping of dense reward functions based on large language models (LLMs). Given a goal described in natural language, Text2Reward generates shaped dense reward functions as an executable program grounded in a compact representation of the environment. Unlike inverse RL and recent work that uses LLMs to write sparse reward codes or unshaped dense rewards with a constant function across timesteps, Text2Reward produces interpretable, free-form dense reward codes that cover a wide range of tasks, utilize existing packages, and allow iterative refinement with human feedback. We evaluate Text2Reward on two robotic manipulation benchmarks (ManiSkill2, MetaWorld) and two locomotion environments of MuJoCo. On 13 of the 17 manipulation tasks, policies trained with generated reward codes achieve similar or better task success rates and convergence speed than expert-written reward codes. For locomotion tasks, our method learns six novel locomotion behaviors with a success rate exceeding 94%. Furthermore, we show that the policies trained in the simulator with our method can be deployed in the real world. Finally, Text2Reward further improves the policies by refining their reward functions with human feedback. Video results are available at https://text-to-reward.github.io/ .

Key facts. Cited by 160 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 61 references that are in the map. First posted on arXiv 20 Sep 2023.

Builds on

51 of 61 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  2. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  4. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Segment Anythingcited by 8,234
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
34 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  11. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  13. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  14. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  15. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  16. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  17. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  18. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  19. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  20. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  21. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  22. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  23. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  24. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  25. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  26. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  27. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 168
  28. 2022DocPrompting: Generating Code by Retrieving the Docscited by 153
  29. 2022Natural Language to Code Translation with Executioncited by 124
  30. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 112
  31. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  32. 2022Predictive Sampling: Real-time Behaviour Synthesis with MuJoCocited by 47
  33. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  34. 2013Credal Networks under Maximum Entropycited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 24; 2025: 49; 2026: 86.

160 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  4. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  5. 2025Self-Adapting Language Modelscited by 84
  6. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 60
  7. 2025RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learningcited by 54
  8. 2026ASPIRE: Agentic /Skills Discovery for Roboticscited by 53
  9. 2025Contextual Experience Replay for Self-Improvement of Language Agentscited by 48
  10. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 38
  11. 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real Worldcited by 38
  12. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
38 more
  1. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  2. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 30
  3. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 30
  4. 2026Aligning Language Models from User Interactionscited by 28
  5. 2024REvolve: Reward Evolution with Large Language Models using Human Feedbackcited by 23
  6. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedbackcited by 22
  7. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  8. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 20
  9. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  10. 2025A Survey on Large Language Models for Automated Planningcited by 17
  11. 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learningcited by 17
  12. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 16
  13. 2024DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulationcited by 15
  14. 2024Language-Guided Object-Centric Diffusion Policy for Generalizable and Collision-Aware Robotic Manipulationcited by 14
  15. 2025LAMARL: LLM-Aided Multi-Agent Reinforcement Learning for Cooperative Policy Generationcited by 13
  16. 2024HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learningcited by 11
  17. 2025Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learningcited by 11
  18. 2024Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewardscited by 9
  19. 2025LERO: LLM-driven Evolutionary framework with Hybrid Rewards and Enhanced Observation for Multi-Agent Reinforcement Learningcited by 9
  20. 2025GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skillcited by 9
  21. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 9
  22. 2025GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learningcited by 9
  23. 2024Text-Aware Diffusion for Policy Learningcited by 8
  24. 2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8
  25. 2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
  26. 2024Tulip Agent -- Enabling LLM-Based Agents to Solve Tasks Using Large Tool Librariescited by 7
  27. 2024RLZero: Direct Policy Inference from Language Without In-Domain Supervisioncited by 7
  28. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrationscited by 7
  29. 2026AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agentscited by 7
  30. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  31. 2024Evaluating Vision-Language Models as Evaluators in Path Planningcited by 6
  32. 2025ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Taskscited by 6
  33. 2025Reward Evolution with Graph-of-Thoughts: A Bi-Level Language Model Framework for Reinforcement Learningcited by 6
  34. 2026Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoningcited by 6
  35. 2024CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learningcited by 5
  36. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5
  37. 2025RoboMoRe: LLM-based Robot Co-design via Joint Optimization of Morphology and Rewardcited by 5
  38. 2025EgoExo-Con: Exploring View-Invariant Video Temporal Understandingcited by 5

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation
  2. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Models
  3. 2024Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution
  4. 2026Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models
  5. 2023Eureka: Human-Level Reward Design via Coding Large Language Models
  6. 2024Dense Reward for Free in Reinforcement Learning from Human Feedback
  7. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
  8. 2025ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations