mlsanityData to 14 Oct 2026

Language to Rewards for Robotic Skill Synthesis

arXiv 2306.08647 ↗First posted 14 Jun 2023cs.RO cs.AI cs.LGcited by 312

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing. Robotics researchers have also explored using LLMs to advance the capabilities of robotic control. However, since low-level robot actions are hardware-dependent and underrepresented in LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human-engineered control primitives to interface with the robot. On the other hand, reward functions are shown to be flexible representations that can be optimized for control policies to achieve diverse tasks, while their semantic richness makes them suitable to be specified by LLMs. In this work, we introduce a new paradigm that harnesses this realization by utilizing LLMs to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions. Meanwhile, combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive behavior creation experience where users can immediately observe the results and provide feedback to the system. To systematically evaluate the performance of our proposed method, we designed a total of 17 tasks for a simulated quadruped robot and a dexterous manipulator robot. We demonstrate that our proposed method reliably tackles 90% of the designed tasks, while a baseline using primitive skills as the interface with Code-as-policies achieves 50% of the tasks. We further validated our method on a real robot arm where complex manipulation skills such as non-prehensile pushing emerge through our interactive system.

Key facts. Cited by 312 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 82 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

50 of 82 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
37 more
  1. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  2. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  4. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  5. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  6. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  7. 2023Reward Design with Language Modelscited by 261
  8. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  9. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  10. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 161
  11. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 150
  12. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118
  13. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  14. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  15. 2020Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Compositioncited by 96
  16. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  17. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  18. 2022Correcting Robot Plans with Natural Language Feedbackcited by 69
  19. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  20. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 63
  21. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  22. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  23. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  24. 2022Inferring Rewards from Language in Contextcited by 50
  25. 2020ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulationcited by 49
  26. 2022Predictive Sampling: Real-time Behaviour Synthesis with MuJoCocited by 47
  27. 2023Barkour: Benchmarking Animal-level Agility with Quadruped Robotscited by 41
  28. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  29. 2022OPT-Mimic: Imitation of Optimized Trajectories for Dynamic Quadruped Behaviorscited by 35
  30. 2023Learning and Adapting Agile Locomotion Skills by Transferring Experiencecited by 33
  31. 2019Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learningcited by 32
  32. 2021LILA: Language-Informed Latent Actionscited by 27
  33. 2023Large Language Models are Built-in Autoregressive Search Enginescited by 27
  34. 2020Learning abstract structure for drawing by efficient motor program inductioncited by 23
  35. 2022Context-Aware Language Modeling for Goal-Oriented Dialogue Systemscited by 22
  36. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformerscited by 17
  37. 2022Sequence-of-Constraints MPC: Reactive Timing-Optimal Control of Sequential Manipulationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 40; 2024: 125; 2025: 91; 2026: 56.

312 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  3. 2023A Comprehensive Overview of Large Language Modelscited by 648
  4. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  5. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  6. 2024Automated Design of Agentic Systemscited by 407
  7. 2024Agent Workflow Memorycited by 383
  8. 2025π*0.6: a VLA That Learns From Experiencecited by 303
  9. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  10. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  11. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  12. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
38 more
  1. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 160
  2. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  3. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  4. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  5. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 129
  6. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 125
  7. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  8. 2025Inducing Programmatic Skills for Agentic Taskscited by 111
  9. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 109
  10. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  11. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  12. 2024Vision Language Models are In-Context Value Learnerscited by 90
  13. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
  14. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 88
  15. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 83
  16. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Roboticscited by 74
  17. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  18. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 68
  19. 2023Vision-Language Models as a Source of Rewardscited by 66
  20. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 64
  21. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 61
  22. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  23. 2023Language Models as Zero-Shot Trajectory Generatorscited by 60
  24. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 60
  25. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 60
  26. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 56
  27. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56
  28. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 54
  29. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 54
  30. 2026ASPIRE: Agentic /Skills Discovery for Roboticscited by 53
  31. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 52
  32. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 51
  33. 2023Prompt a Robot to Walk with Large Language Modelscited by 45
  34. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 45
  35. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  36. 2023Creative Robot Tool Use with Large Language Modelscited by 43
  37. 2024Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Followingcited by 42
  38. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 42

50 most cited of 312 All 312 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Can only LLMs do Reasoning?: Potential of Small Language Models in Task Planning
  2. 2023Conditionally Combining Robot Skills using Large Language Models
  3. 2024Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs
  4. 2025Collision- and Reachability-Aware Multi-Robot Control with Grounded LLM Planners
  5. 2025LLM-Guided Task- and Affordance-Level Exploration in Reinforcement Learning
  6. 2026Hybrid Framework for Robotic Manipulation: Integrating Reinforcement Learning and Large Language Models
  7. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
  8. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics