mlsanityData to 14 Oct 2026

REvolve: Reward Evolution with Large Language Models using Human Feedback

arXiv 2406.01309 ↗First posted 3 Jun 2024cs.NE cs.AIcited by 23

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Designing effective reward functions is crucial to training reinforcement learning (RL) algorithms. However, this design is non-trivial, even for domain experts, due to the subjective nature of certain tasks that are hard to quantify explicitly. In recent works, large language models (LLMs) have been used for reward generation from natural language task descriptions, leveraging their extensive instruction tuning and commonsense understanding of human behavior. In this work, we hypothesize that LLMs, guided by human feedback, can be used to formulate reward functions that reflect human implicit knowledge. We study this in three challenging settings -- autonomous driving, humanoid locomotion, and dexterous manipulation -- wherein notions of ``good" behavior are tacit and hard to quantify. To this end, we introduce REvolve, a truly evolutionary framework that uses LLMs for reward design in RL. REvolve generates and refines reward functions by utilizing human feedback to guide the evolution process, effectively translating implicit human knowledge into explicit reward functions for training (deep) RL agents. Experimentally, we demonstrate that agents trained on REvolve-designed rewards outperform other state-of-the-art baselines.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 74 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

42 of 74 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Models19 Oct
  2. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learning20 Sep
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 38

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Segment Anythingcited by 8,234
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  12. 2015Prioritized Experience Replaycited by 1,099
26 more
  1. 2023Large Language Models as Optimizerscited by 864
  2. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  3. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  4. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  6. 2023Instruction Tuning with GPT-4cited by 686
  7. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 523
  8. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  9. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  10. 2016Cooperative Inverse Reinforcement Learningcited by 300
  11. 2023Reward Design with Language Modelscited by 261
  12. 2017Inverse Reward Designcited by 249
  13. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 240
  14. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 193
  15. 2024Large Language Models to Enhance Bayesian Optimizationcited by 135
  16. 2019Optimal Policies Tend to Seek Powercited by 119
  17. 2021Reward (Mis)design for Autonomous Drivingcited by 104
  18. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 103
  19. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 87
  20. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  21. 2023How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligencecited by 58
  22. 2023SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledgecited by 49
  23. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  24. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?cited by 45
  25. 2018Intervention Aided Reinforcement Learning for Safe and Practical Policy Optimization in Navigationcited by 13
  26. 2022Sim2real for Autonomous Vehicle Control using Executable Digital Twincited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 14.

23 papers.

Sort:

By area:

  1. 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-benchcited by 71
  2. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 51
  3. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Spacecited by 43
  4. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 20
  5. 2024Can Large Language Models Reason? A Characterization via 3-SATcited by 12
  6. 2025Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policiescited by 8
  7. 2025LLM-Guided Search for Deletion-Correcting Codescited by 4
  8. 2026RF-Agent: Automated Reward Function Design via Language Agent Tree Searchcited by 4
  9. 2026ScienceFlow: A long-horizon agent for ML research, scientific discovery and beyondcited by 4
  10. 2025DriveMind: A Dual Visual Language Model-based Reinforcement Learning Framework for Autonomous Drivingcited by 3
  11. 2025Automated Reward Design for Gran Turismocited by 3
  12. 2026HARBOR: A Harness Framework for Agentic Robot Reinforcement Learningcited by 3
11 more
  1. 2026MEMENTO: Memory-Guided Memetic Code-as-Policy Evolutioncited by 3
  2. 2025Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation with Large Language Modelscited by 2
  3. 2026Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Modelscited by 1
  4. 2026The Past Is Not Past: Memory-Enhanced Dynamic Reward Shapingcited by 1
  5. 2026RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypothesescited by 1
  6. 2026Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Designcited by 1
  7. 2026Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligencecited by 1
  8. 2026Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learningcited by 0
  9. 2026Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasonerscited by 0
  10. 2026Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimizationcited by 0
  11. 2026A Bird's-Eye View of Iterative Reward Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  2. 2024Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution
  3. 2025Reward Evolution with Graph-of-Thoughts: A Bi-Level Language Model Framework for Reinforcement Learning
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  5. 2026Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
  6. 2026The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination
  7. 2023The History and Risks of Reinforcement Learning and Human Feedback
  8. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning