mlsanityData to 14 Oct 2026

EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning

arXiv 2609.12459 ↗First posted 11 Sep 2026cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Open-ended reinforcement learning often relies on rubric-based rewards for tasks without directly verifiable answers. Yet the policy and reward system form a dynamic feedback loop: as the policy optimizes the current reward, an initially useful reward system may become unreliable due to reward hacking or reduced response discriminability. The reward system should therefore evolve rather than remain fixed during training. Existing dynamic-rubric methods adapt evaluation criteria, but reward failures can also arise from scoring mechanisms or signal composition. We introduce EvoRS, a self-evolving RL framework that evolves the reward system from on-policy experience, representing it as an executable Reward-DAG. Specifically, an agentic designer updates this system from on-policy rollouts and reward traces to maintain train-time reliability. Across writing and roleplay, EvoRS achieves the best quality under all three judges, outperforming the policy by (2.107) and (4.767) points, respectively, while reducing reward hacking and coverage failures and preserving reward informativeness. Ablations confirm that a comprehensive fixed reward system cannot remain reliable in open-ended tasks and must evolve throughout training.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 52 of its 97 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

52 of 97 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026ProRL: Effective Reinforcement Learning for Proactive Recommendation via Rectified Policy Gradient Estimation27 May
  2. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric15 Feb
  3. 2025Reinforcement Learning with Rubric Anchors18 Aug
  4. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains23 Jul

Other references · 48

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2024A Survey on LLM-as-a-Judgecited by 1,423
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  6. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  7. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  8. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  9. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  10. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  11. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  12. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
36 more
  1. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  2. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  3. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  4. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  5. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  6. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  7. 2026Self-Harness: Harnesses That Improve Themselvescited by 107
  8. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  9. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  10. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  11. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  12. 2025One Token to Fool LLM-as-a-Judgecited by 60
  13. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 55
  14. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  15. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 53
  16. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 50
  17. 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writingcited by 42
  18. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 39
  19. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Usecited by 31
  20. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 28
  21. 2026CoEvolve: Training LLM Agents via Agent-Data Mutual Evolutioncited by 23
  22. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimizationcited by 21
  23. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Modelscited by 19
  24. 2024Fine-Tuning Language Models with Reward Learning on Policycited by 15
  25. 2026ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RLcited by 14
  26. 2026EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learningcited by 14
  27. 2026EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learningcited by 14
  28. 2026EnvHarness: Awakening Static Worlds for Agent Learningcited by 13
  29. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimizationcited by 11
  30. 2026HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playingcited by 10
  31. 2026No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learningcited by 8
  32. 2024LINKAGE: Listwise Ranking among Varied-Quality References for Non-Factoid QA Evaluation via LLMscited by 5
  33. 2026From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoningcited by 5
  34. 2026Automatically Finding Reward Model Biasescited by 4
  35. 2026Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learningcited by 3
  36. 2026Permutation-Consensus Listwise Judging for Robust Factuality Evaluationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Verifier-Harness Co-Evolution for Training-Free Self-Improvement on Open-Ended Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RewardWeaver: Long-Horizon Interactive Learning for Language Agents via Self-Evolving Reward Adaptation
  2. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
  3. 2026ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning
  4. 2026SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
  5. 2026EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation
  6. 2026RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution
  7. 2026Reward Hacking in Rubric-Based Reinforcement Learning
  8. 2026From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL