mlsanityData to 14 Oct 2026

ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents

arXiv 2606.21262 ↗First posted 19 Jun 2026cs.AI cs.CLcited by 4

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning for multi-step LLM agents often relies on scalar rewards that indicate success but cannot explain why a trajectory is good or bad. Rubric-based rewards improve interpretability through natural-language criteria, but existing methods share two limitations: they score at the trajectory level, offering no guidance for individual steps; and their scorer is closed-source and static, so it cannot adapt as the agent evolves during training. We propose ARCO (Adaptive Rubric CO-evolution), which generates a per-step rubric and predicts a rubric-conditioned step-level reward for each action, and continually updates this rubric model on on-policy rollouts so that its criteria and scores co-evolve with the agent's improving behavior. Across HotpotQA, 2WikiMultiHopQA, and MuSiQue with two open-source backbones, ARCO achieves the highest EM in all settings over outcome-, rubric-, and process-reward baselines, and analyses show its rubrics are step-specific, robust to design choices, and useful for diagnosing agent behavior. Code and data are available at https://github.com/zihangtian/ARCO.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 35 of its 40 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

35 of 40 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Co-Evolution of Policy and Internal Reward for Language Agents3 Apr
  2. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System2 Feb
  3. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress11 Nov
  4. 2025Reinforcement Learning for Long-Horizon Multi-Turn Search Agents28 Oct
  5. 2025Agentic Reinforcement Learning with Implicit Step Rewards23 Sep
  6. 2025RRO: LLM Agent Optimization Through Rising Reward Trajectories27 May
  7. 2025Process Reward Models for LLM Agents: Practical Framework and Directions14 Feb

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
16 more
  1. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  2. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  3. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  4. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  5. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 114
  6. 2025Reinforcement Learning with Rubric Anchorscited by 113
  7. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 88
  8. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 88
  9. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  10. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 53
  11. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
  12. 2025R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learningcited by 34
  13. 2024CARMO: Dynamic Criteria Generation for Context-Aware Reward Modellingcited by 22
  14. 2024Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolutioncited by 5
  15. 2025Mutual-Taught for Co-adapting Policy and Reward Modelscited by 3
  16. 2026HAPS: Hierarchical LLM Routing with Joint Architecture and Parameter Searchcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Gradingcited by 31
  2. 2026DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Spacecited by 2
  3. 2026ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signalscited by 1
  4. 2026Self-Designed Evaluators and Warm Memory for Long-Horizon Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
  2. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation
  3. 2026ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning
  4. 2026ARISE: Adapting to Evolving Capability Gaps in Agentic Reinforcement Learning
  5. 2026SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning
  6. 2026SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
  7. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
  8. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape