mlsanityData to 7 Oct 2026

Training Advisors for LLM Agents from Task Outcomes

arXiv 2610.09858 ↗First posted 7 Oct 2026cs.CL cs.AI cs.LGcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents also fits: General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language model agents tackle multi-step tasks by interleaving reasoning and tool calls with observations from the environment. Prior work has shown that natural-language feedback can help these agents revise their decisions during task execution. We introduce Caddie, a method for training critics to provide natural-language analysis and advice as agents work through a task. Unlike approaches that rely on step-level labels or reference critiques, Caddie learns from whether the agent ultimately succeeds after receiving the critic's feedback. We optimize the critic through reinforcement learning while keeping the base model frozen. Trained on multi-hop question answering with a single base model, our Qwen3-4B critic improves success rates across four base models of different scales and architectures, including three not used during critic training. On the MuSiQue benchmark, the trained critic improves Qwen3-4B's success rate by more than 25 percentage points, surpassing the performance of Kimi K3 without a critic. The same critic also yields gains on out-of-domain interactive benchmarks, including τ3 and DeepDive, with no additional training. Our results show that agents can decide when to seek help from a critic at inference time and that outcome-based critic training can produce guidance that transfers across base models and task domains.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 12 of its 34 references that are in the map. First posted on arXiv 7 Oct 2026.

Builds on

12 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry4 Jul
  2. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress11 Nov

Other references · 10

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  5. 2022Generating Sequences by Learning to Self-Correctcited by 271
  6. 2025DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RLcited by 53
  7. 2025How to Train Your Advisor: Steering Black-Box LLMs with Advisor Modelscited by 20
  8. 2026ICRL: Learning to Internalize Self-Critique with Reinforcement Learningcited by 3
  9. 2026Steer, Don't Solve: Training Small Critic Models for Large Code Agentscited by 3
  10. 2026Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement
  2. 2026CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents
  3. 2026SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories
  4. 2025Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space
  5. 2024Training Agents with Weakly Supervised Feedback from Large Language Models
  6. 2025Reinforce LLM Reasoning through Multi-Agent Reflection
  7. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
  8. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning