mlsanityData to 7 Oct 2026

StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction

arXiv 2605.06642 ↗First posted 7 May 2026cs.CL cs.AIcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large language models (LLMs) are increasingly used as interactive agents, but optimizing them for long-horizon decision making remains difficult because current methods are largely purely reactive, which weakens both exploration and credit assignment over extended trajectories. In this work, we present Strategic Trajectory Abstraction (StraTA), a simple framework that introduces an explicit trajectory-level strategy into agentic reinforcement learning (RL). StraTA samples a compact strategy from the initial task state, conditions subsequent actions on that strategy, and trains strategy generation and action execution jointly with a hierarchical GRPO-style rollout design, further enhanced by diverse strategy rollout and critical self-judgment. Experiments on ALFWorld, WebShop, and SciWorld show that StraTA consistently improves both sample efficiency and final performance over strong baselines. StraTA reaches success rates of 93.1% on ALFWorld and 84.2% on WebShop. On SciWorld, StraTA attains a 63.5% overall score, outperforming frontier closed-source models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 60 of its 68 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

60 of 68 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  2. 2026HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents18 Feb
  3. 2025Tree Search for LLM Agent Reinforcement Learning25 Sep
  4. 2025Agentic Reinforcement Learning with Implicit Step Rewards23 Sep
  5. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning10 Sep
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey2 Sep
  7. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learning5 Aug
  8. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  9. 2025Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning1 May
  10. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  11. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May
  12. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb
2 more
  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  12. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
34 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  4. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2024A Survey on LLM-as-a-Judgecited by 1,384
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  12. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  13. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  14. 2025Group Sequence Policy Optimizationcited by 788
  15. 2024Self-Rewarding Language Modelscited by 654
  16. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  17. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  18. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  19. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  20. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  21. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  22. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  23. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  24. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  25. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  26. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  27. 2025Learning to Reason without External Rewardscited by 199
  28. 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Taskscited by 190
  29. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  30. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
  31. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Modelscited by 93
  32. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectoriescited by 56
  33. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  34. 2025Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflowscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learningcited by 0
  2. 2026Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning
  2. 2025Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents
  3. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
  4. 2026Expanding LLM Agent Boundaries with Strategy-Guided Exploration
  5. 2026RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
  6. 2026Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe
  7. 2026Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
  8. 2026Rethinking Agentic Reinforcement Learning In Large Language Models