mlsanityData to 7 Oct 2026

TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agents

arXiv 2602.11767 ↗First posted 12 Feb 2026cs.AI cs.CL cs.LGcited by 6

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Advances in large language models (LLMs) are driving a shift toward using reinforcement learning (RL) to train agents from iterative, multi-turn interactions across tasks. However, multi-turn RL remains challenging as rewards are often sparse or delayed, and environments can be stochastic. In this regime, naive trajectory sampling can hinder exploitation and induce mode collapse. We propose TSR (Trajectory-Search Rollouts), a training-time approach that repurposes test-time scaling ideas for improved per-turn rollout generation. TSR performs lightweight tree-style search to construct higher-quality trajectories by selecting promising actions and trajectory prefixes during rollout generation. This improves rollout quality while preserving stable policy optimization and remains compatible with standard policy-gradient optimizers by design. Across Sokoban, FrozenLake, and WebShop, TSR achieves success-rate gains of up to 15 percentage points and converges in fewer optimization steps, while trading additional training-time rollout compute for stronger policies that require no search at inference time. By moving search from test time to the rollout stage of training, TSR provides a modular mechanism for stronger multi-turn agent learning, complementary to existing frameworks and rejection-sampling-style selection methods.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 47 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

33 of 47 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent20 Nov
  2. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress11 Nov
  3. 2025Tree Search for LLM Agent Reinforcement Learning25 Sep
  4. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learning5 Aug
  5. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment17 May
  6. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  7. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  8. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks19 Mar
  9. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  10. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  12. 2025Process Reinforcement through Implicit Rewardscited by 478
12 more
  1. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  2. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  3. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  4. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  5. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  6. 2025Scaling Test-time Compute for LLM Agentscited by 59
  7. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  8. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  9. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Trainingcited by 33
  10. 2025Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewardscited by 12
  11. 2025Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Controlcited by 2
  12. 2024On shallow planning under partial observabilitycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Modelscited by 3
  2. 2026Process Reward Informed Tree Rollout for Effective Multi-Turn RLcited by 1
  3. 2026Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Biascited by 0
  4. 2026MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agentscited by 0
  5. 2026Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agentscited by 0
  6. 2026SIPO: Selective-Inference Policy Optimization for Tree-Structured Agentic RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
  2. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents
  3. 2026RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
  4. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  5. 2026Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agents
  6. 2025Reinforcement Learning for Long-Horizon Multi-Turn Search Agents
  7. 2026RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents
  8. 20263SPO: State-Score-Supervised Policy Optimization for LLM Agents