mlsanityData to 7 Oct 2026

T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning

arXiv 2610.00388 ↗First posted 30 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning enables large language model (LLM) agents to learn multi-step behaviors through interaction with their environments. However, rewards in many interactive tasks reflect only the final outcome, providing limited guidance on which intermediate decisions advance the task. Successful training trajectories contain intermediate states that can provide supervision for subsequent interactions. We introduce Trajectory-to-Step Policy Optimization (T2SPO), a method that uses past interaction trajectories to provide step-level feedback for policy learning. T2SPO derives remaining-distance targets from successful trajectories and pairs them with representations of the states visited along the way. Conditioned on these examples, a pretrained TabPFN regressor estimates the remaining distance to success at each state of a new rollout. Changes in this distance estimate across consecutive states yield auxiliary credit for agent steps alongside task-level supervision. As training proceeds, newly completed trajectories refresh the estimator's context, incorporating new experience without updating its parameters. Experiments with 1.5B and 7B language models on ALFWorld and WebShop show that T2SPO consistently improves overall task success over GRPO.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 25 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

21 of 25 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs18 Dec
  2. 2025Agentic Reinforced Policy Optimization26 Jul
  3. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  4. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  5. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  12. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
4 more
  1. 2021Transformers Can Do Bayesian Inferencecited by 338
  2. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
  3. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  4. 2026V0.5: Generalist Value Model as a Prior for Sparse RL Rolloutscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agents
  2. 2026TAPO: Transition-Aware Policy Optimization for LLM Agents
  3. 2026From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training
  4. 2025STeCa: Step-level Trajectory Calibration for LLM Agent Learning
  5. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
  6. 2026STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training
  7. 20263SPO: State-Score-Supervised Policy Optimization for LLM Agents
  8. 2026Learning Agentic Policy from Action Guidance