mlsanityData to 14 Oct 2026

Dual-Evolving Reinforcement Learning for Automated Reward Design

arXiv 2512.13399 ↗First posted 15 Dec 2025cs.AI cs.CLcited by 1

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Crafting effective reward signals remains a central challenge in reinforcement learning (RL) for complex reasoning. Existing methods for automated reward optimization search with a fixed proposer which never learns from its costly trials. We present Dual-Evolving Reinforcement Learning (DERL), a dual-training framework for autonomous discovery of reward structures. An outer-loop proposer composes a reward from task primitives, and an inner policy trains with this reward. The proposer itself learns by policy gradient from the validation performance of the inner policy. Thus, DERL utilizes past-trial feedback to shape its reward-proposal distribution, an empirical meta-gradient of task success. We validate DERL on embodied (ALFWorld), scientific (ScienceWorld), and mathematical (GSM8K, MATH) reasoning. On multi-turn agent benchmarks, DERL outperforms strong baselines, significantly on out-of-distribution tasks. In mathematics, DERL improves on the outcome reward where directly added primitives hurt. Further analyses under matched compute show that our proposer effectively captures the meta-gradient to discover useful reward signals.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 45 of its 55 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

45 of 55 references.

References · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  10. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  11. 2016Concrete Problems in AI Safetycited by 1,642
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
33 more
  1. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 867
  2. 2016Learning to learn by gradient descent by gradient descentcited by 620
  3. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  4. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  5. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  6. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  7. 2017Population Based Training of Neural Networkscited by 367
  8. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  9. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  10. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  11. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  12. 2025A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systemscited by 183
  13. 2018Meta-Gradient Reinforcement Learningcited by 159
  14. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 150
  15. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 135
  16. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 114
  17. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  18. 2020Discovering Reinforcement Learning Algorithmscited by 77
  19. 2025RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agentscited by 58
  20. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  21. 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
  22. 2024G"odel Agent: A Self-Referential Agent Framework for Recursive Self-Improvementcited by 29
  23. 2024Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environmentscited by 27
  24. 2024QueryAgent: A Reliable and Efficient Reasoning Framework with Environmental Feedback-based Self-Correctioncited by 27
  25. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  26. 2025Evolution Strategies at the Hyperscalecited by 25
  27. 2024Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Modelscited by 15
  28. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selectioncited by 12
  29. 2025Re:Form -- Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafnycited by 10
  30. 2025Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifiescited by 8
  31. 2023MarkQA: A large scale KBQA dataset with numerical reasoningcited by 6
  32. 2024TARGA: Targeted Synthetic Data Generation for Practical Reasoning over Structured Datacited by 5
  33. 2025Automated Reward Design for Gran Turismocited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026X-Tree: Tokenizing Reusable Experience for Efficient Agent Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023BiERL: A Meta Evolutionary Reinforcement Learning Framework via Bilevel Optimization
  2. 2026Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Models
  3. 2024Ancestral Reinforcement Learning: Unifying Zeroth-Order Optimization and Genetic Algorithms for Reinforcement Learning
  4. 2025SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents
  5. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms
  6. 2024Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
  7. 2026EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning
  8. 2026EVOM: Agentic Meta-Evolution of Actor-Critic Architectures for Reinforcement Learning