mlsanityData to 14 Oct 2026

DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter

arXiv 2610.17666 ↗First posted 13 Oct 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps. Within-group step comparisons depend on reliable state matching and sufficiently diverse sampled outcomes, while direct step scoring may require costly external evaluators, human annotations, or task-specific heuristics. We introduce DeltaTTG to learn a reusable progress signal through a shared progress meter, using low-cost supervision from diverse policies' recorded trajectories without task-specific progress rules. Its supervision target, Token-to-Go (TTG), the number of tokens remaining to be generated until task completion, serves as an observable proxy for remaining completion effort. During actor training, changes in the frozen meter's estimates redistribute outcome-derived advantages across steps without altering the reward, preserving each trajectory's advantage sign and token-average scale. On ALFWorld, PaperSearchQA, and BFCLv3, DeltaTTG improves the average score over GRPO by 5.7 and 7.2 points with Qwen3-4B and Llama-3.1-8B-Instruct, respectively, outperforms all compared RL baselines on average, and keeps actor-training GPU-hours close to GRPO. Further experiments show improved sample efficiency and out-of-domain performance, and support the meter's ability to distinguish step quality and guide test-time action selection across actor sizes. The code is available at https://github.com/Xuyan923r/DeltaTTG.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

37 of 50 references.

References · 37

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  10. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  11. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  12. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
25 more
  1. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  2. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  3. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  4. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  5. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  6. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 200
  7. 2025Where LLM Agents Fail and How They can Learn From Failurescited by 133
  8. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 109
  9. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 97
  10. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 85
  11. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 73
  12. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 66
  13. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
  14. 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewardscited by 42
  15. 2026Milestone-Guided Policy Learning for Long-Horizon Language Agentscited by 19
  16. 2026Verified Critical Step Optimization for LLM Agentscited by 14
  17. 2026TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agentscited by 7
  18. 2026DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolutioncited by 6
  19. 2026PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVRcited by 6
  20. 2026Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Modelscited by 4
  21. 2026ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignmentcited by 3
  22. 2026P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answeringcited by 2
  23. 2026Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flatteningcited by 2
  24. 2026Inducing Process Supervision from Outcome-Only Reinforcement Learningcited by 1
  25. 2026Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Trainingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
  2. 2026T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
  3. 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
  4. 2026Retrospective Progress-Aware Self-Refinement for LLM Agent Training
  5. 2026StepOPSD: Step-Aware Online Preference Self-Distillation for Agent Reinforcement Learning
  6. 2026HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents
  7. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
  8. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents