mlsanityData to 7 Oct 2026

Eureka: Human-Level Reward Design via Coding Large Language Models

arXiv 2310.12931 ↗First posted 19 Oct 2023cs.RO cs.AI cs.LGcited by 542

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. However, harnessing them to learn complex low-level manipulation tasks, such as dexterous pen spinning, remains an open problem. We bridge this fundamental gap and present Eureka, a human-level reward design algorithm powered by LLMs. Eureka exploits the remarkable zero-shot generation, code-writing, and in-context improvement capabilities of state-of-the-art LLMs, such as GPT-4, to perform evolutionary optimization over reward code. The resulting rewards can then be used to acquire complex skills via reinforcement learning. Without any task-specific prompting or pre-defined reward templates, Eureka generates reward functions that outperform expert human-engineered rewards. In a diverse suite of 29 open-source RL environments that include 10 distinct robot morphologies, Eureka outperforms human experts on 83% of the tasks, leading to an average normalized improvement of 52%. The generality of Eureka also enables a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating. Finally, using Eureka rewards in a curriculum learning setting, we demonstrate for the first time, a simulated Shadow Hand capable of performing pen spinning tricks, adeptly manipulating a pen in circles at rapid speed.

Key facts. Cited by 542 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 75 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

46 of 75 references.

References · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2016Generative Adversarial Imitation Learningcited by 1,576
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  12. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
34 more
  1. 2017Deep Reinforcement Learning that Matterscited by 727
  2. 2019Solving Rubik's Cube with a Robot Handcited by 699
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  4. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  5. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  7. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  8. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  9. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  10. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  11. 2023Reward Design with Language Modelscited by 257
  12. 2017Inverse Reward Designcited by 246
  13. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  14. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  15. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  16. 2023Language-Driven Representation Learning for Roboticscited by 163
  17. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  18. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  19. 2023Vision-Language Models as Success Detectorscited by 138
  20. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  21. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  22. 2022Evolution through Large Modelscited by 131
  23. 2022Rapid Locomotion via Reinforcement Learningcited by 130
  24. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  25. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  26. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  27. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  28. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  29. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  30. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  31. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  32. 2022Predictive Sampling: Real-time Behaviour Synthesis with MuJoCocited by 39
  33. 2023Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thoughtcited by 39
  34. 2019Evolving Rewards to Automate Reinforcement Learningcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 175; 2025: 162; 2026: 196.

542 papers.

Sort:

By area:

  1. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  2. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  3. 2024Automated Design of Agentic Systemscited by 360
  4. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  5. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  7. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  8. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 266
  9. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 258
  10. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  11. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  12. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
38 more
  1. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  2. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  3. 2026Towards End-to-End Automation of AI Researchcited by 150
  4. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
  5. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  6. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  7. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  8. 2025Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memorycited by 129
  9. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  10. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  11. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  13. 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinementcited by 124
  14. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 120
  15. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  16. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  17. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  18. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  19. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  20. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  21. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 99
  22. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  23. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  24. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  25. 2024Vision Language Models are In-Context Value Learnerscited by 84
  26. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  27. 2024LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discoverycited by 82
  28. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  29. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 72
  30. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  31. 2023Vision-Language Models as a Source of Rewardscited by 65
  32. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  33. 2026AutoHarness: improving LLM agents by automatically synthesizing a code harnesscited by 63
  34. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 60
  35. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  36. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 59
  37. 2024Efficient Evolutionary Search Over Chemical Space with Large Language Modelscited by 59
  38. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55

50 most cited of 542 All 542 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learning
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2024Prototypical Reward Network for Data-Efficient RLHF
  4. 2026Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
  5. 2025Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping
  6. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
  7. 2026RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards ↗
  8. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics