mlsanityData to 7 Oct 2026

A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning

arXiv 2410.14660 ↗First posted 18 Oct 2024cs.LGcited by 29

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Large Language Models (LLMs) have shown significant potential in designing reward functions for Reinforcement Learning (RL) tasks. However, obtaining high-quality reward code often involves human intervention, numerous LLM queries, or repetitive RL training. To address these issues, we propose CARD, a LLM-driven Reward Design framework that iteratively generates and improves reward function code. Specifically, CARD includes a Coder that generates and verifies the code, while a Evaluator provides dynamic feedback to guide the Coder in improving the code, eliminating the need for human feedback. In addition to process feedback and trajectory feedback, we introduce Trajectory Preference Evaluation (TPE), which evaluates the current reward function based on trajectory preferences. If the code fails the TPE, the Evaluator provides preference feedback, avoiding RL training at every iteration and making the reward function better aligned with the task objective. Empirical results on Meta-World and ManiSkill2 demonstrate that our method achieves an effective balance between task performance and token efficiency, outperforming or matching the baselines across all tasks. On 10 out of 12 tasks, CARD shows better or comparable performance to policies trained with expert-designed rewards, and our method even surpasses the oracle on 3 tasks.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 69 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

43 of 69 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Models19 Oct
  2. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learning20 Sep
  3. 2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation6 Jun
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2018Learning Dexterous In-Hand Manipulationcited by 643
  10. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  11. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
25 more
  1. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  2. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  3. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  4. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  5. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  6. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  7. 2023Reward Design with Language Modelscited by 257
  8. 2017Inverse Reward Designcited by 246
  9. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  10. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  11. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  12. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  13. 2023Language-Driven Representation Learning for Roboticscited by 163
  14. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  15. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  16. 2023Vision-Language Models as Success Detectorscited by 138
  17. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  18. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  19. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  20. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  21. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 60
  22. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
  23. 2023Optimal Transport for Offline Imitation Learningcited by 42
  24. 2023Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thoughtcited by 39
  25. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 13.

29 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 36
  3. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  4. 2025Dynamic and Generalizable Process Reward Modelingcited by 16
  5. 2025GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skillcited by 9
  6. 2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8
  7. 2025Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policiescited by 8
  8. 2025Reward Evolution with Graph-of-Thoughts: A Bi-Level Language Model Framework for Reinforcement Learningcited by 6
  9. 2026From Generative Engines to Actionable Simulators: The Imperative of Physical Grounding in World Modelscited by 4
  10. 2026SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Groundingcited by 4
  11. 2026Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generationcited by 3
  12. 2026The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordinationcited by 3
17 more
  1. 2025Automated Reward Design for Gran Turismocited by 2
  2. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learningcited by 2
  3. 2025Leveraging LLMs for reward function design in reinforcement learning control taskscited by 2
  4. 2026EvoNav: Evolutionary Reward Function Design for Robot Navigation with Large Language Modelscited by 2
  5. 2025Multilinguality in LLM-Designed Reward Functions for Restless Bandits: Effects on Task Performance and Fairnesscited by 1
  6. 2025An Automated Reinforcement Learning Reward Design Framework with Large Language Model for Cooperative Platoon Coordinationcited by 1
  7. 2025Deep Reinforcement Learning for Dynamic Algorithm Configuration: A Case Study on Optimizing OneMax with the (1+(λ,λ))-GAcited by 1
  8. 2026When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RLcited by 1
  9. 2026Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Designcited by 1
  10. 2025ROAD: Responsibility-Oriented Reward Design for Reinforcement Learning in Autonomous Drivingcited by 0
  11. 2025LinguaFluid: Language Guided Fluid Control via Semantic Rewards in Reinforcement Learningcited by 0
  12. 2026READY: Reward Discovery for Meta-Black-Box Optimizationcited by 0
  13. 2026Adaptive RAN Slicing Control via Reward-Free Self-Finetuning Agentscited by 0
  14. 2026Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learningcited by 0
  15. 2026Generating Robust Portfolios of Optimization Models using Large Language Modelscited by 0
  16. 2026The Rise of Verbal Reinforcement Learningcited by 0
  17. 2026Qwen-Planner-Agent: A Closed-Loop AI-for-AI Framework for Real-World Mobile Planner Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Boosting Universal LLM Reward Design through Heuristic Reward Observation Space Evolution
  2. 2024Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  4. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  5. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Making
  6. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2025A Systematic Analysis of Base Model Choice for Reward Modeling