mlsanityData to 7 Oct 2026

SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks

arXiv 2503.15478 ↗First posted 19 Mar 2025cs.LGcited by 108

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large language model (LLM) agents need to perform multi-turn interactions in real-world tasks. However, existing multi-turn RL algorithms for optimizing LLM agents fail to perform effective credit assignment over multiple turns while leveraging the generalization capabilities of LLMs and it remains unclear how to develop such algorithms. To study this, we first introduce a new benchmark, ColBench, where an LLM agent interacts with a human collaborator over multiple turns to solve realistic tasks in backend programming and frontend design. Building on this benchmark, we propose a novel RL algorithm, SWEET-RL (RL with Step-WisE Evaluation from Training-time information), that uses a carefully designed optimization objective to train a critic model with access to additional training-time information. The critic provides step-level rewards for improving the policy model. Our experiments demonstrate that SWEET-RL achieves a 6% absolute improvement in success and win rates on ColBench compared to other state-of-the-art multi-turn RL algorithms, enabling Llama-3.1-8B to match or exceed the performance of GPT4-o in realistic collaborative content creation.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 41 of its 56 references that are in the map. First posted on arXiv 19 Mar 2025.

Builds on

41 of 56 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
29 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  3. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  4. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  5. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  6. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  7. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  8. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  9. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  10. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  11. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  12. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  13. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  14. 2024Free Process Rewards without Process Labelscited by 162
  15. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  16. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 126
  17. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  18. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  19. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  20. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  21. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  22. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  23. 2024Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewardscited by 31
  24. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  25. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
  26. 2025Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedbackcited by 22
  27. 2019Learning to Manipulate Object Collections Using Grounded State Representationscited by 11
  28. 2024Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planningcited by 7
  29. 2019Attention-Privileged Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 57; 2026: 51.

108 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  4. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 129
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 91
  7. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  8. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
  9. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
  10. 2025Self-Challenging Language Model Agentscited by 56
  11. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  12. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
38 more
  1. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  3. 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websitescited by 43
  4. 2025Training Proactive and Personalized LLM Agentscited by 42
  5. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 40
  6. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  7. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 35
  8. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 29
  9. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  10. 2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
  11. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learningcited by 28
  12. 2025Get Experience from Practice: LLM Agents with Record & Replaycited by 22
  13. 2025SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graphcited by 22
  14. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 19
  15. 2025How to Train Your LLM Web Agent: A Statistical Diagnosiscited by 18
  16. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agentscited by 18
  17. 2025Non-Collaborative User Simulators for Tool Agentscited by 17
  18. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
  19. 2025Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learningcited by 16
  20. 2025A Practitioner's Guide to Multi-turn Agentic Reinforcement Learningcited by 16
  21. 2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agentscited by 15
  22. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  23. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 13
  24. 2025Position: The Real Barrier to LLM Agent Usability is Agentic ROIcited by 12
  25. 2025InteractComp: Evaluating Search Agents With Ambiguous Queriescited by 12
  26. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMscited by 11
  27. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
  28. 2025The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperationcited by 11
  29. 2026ARLArena: A Unified Framework for Stable Agentic Reinforcement Learningcited by 11
  30. 2025Evolving in Tasks: Empowering the Multi-modality Large Language Model as the Computer Use Agentcited by 10
  31. 2025The Alignment Waltz: Jointly Training Agents to Collaborate for Safetycited by 10
  32. 2025RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMscited by 10
  33. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  34. 2025ARIA: Training Language Agents with Intention-Driven Reward Aggregationcited by 9
  35. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 9
  36. 2025SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agentscited by 8
  37. 2026RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulationcited by 8
  38. 2025Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRAcited by 7

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MALT: Improving Reasoning with Multi-Agent LLM Training
  2. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration
  3. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning
  4. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
  5. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
  6. 2025Token-Efficient RL for LLM Reasoning
  7. 2025DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs
  8. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework