mlsanityData to 14 Oct 2026

Can RL Improve Generalization of LLM Agents? An Empirical Study

arXiv 2603.12011 ↗First posted 12 Mar 2026cs.AIcited by 4

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement fine-tuning (RFT) has shown promise for training LLM agents to perform multi-turn decision-making based on environment feedback. However, most existing evaluations remain largely in-domain: training and testing are conducted in the same environment or even on the same tasks. In real-world deployment, agents may operate in unseen environments with different background knowledge, observation spaces, and action interfaces. To characterize the generalization profile of RFT under such shifts, we conduct a systematic study along three axes: (1) within-environment generalization across task difficulty, (2) cross-environment transfer to unseen environments, and (3) sequential multi-environment training to quantify transfer and forgetting. Our results show that RFT generalizes well across task difficulty within an environment, but exhibits weaker transfer to unseen environments, which correlates with shifts in both semantic priors and observation/action interfaces. In contrast, sequential training yields promising downstream gains with minimal upstream forgetting, and mixture training across environments improves the overall balance. We further provide detailed analyses and deeper insights, and hope our work helps the community develop and deploy generalizable LLM agents.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 64 of its 78 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

64 of 78 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch2 Nov
  2. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning10 Sep
  3. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution27 May
  4. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving12 May
  5. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  6. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agents3 Feb
  7. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models14 Sep
  8. 2024AGILE: A Novel Reinforcement Learning Framework of LLM Agents23 May
  9. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May
  10. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents4 Mar
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 52

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
38 more
  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  3. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  4. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  6. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  9. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  10. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  12. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  13. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  14. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  15. 2023Explainability for Large Language Models: A Surveycited by 333
  16. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 287
  17. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  18. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  19. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 225
  20. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
  21. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  22. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
  23. 2024Rule Based Rewards for Language Model Safetycited by 108
  24. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  25. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  26. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 85
  27. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 76
  28. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 76
  29. 2025DeepAgent: A General Reasoning Agent with Scalable Toolsetscited by 71
  30. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54
  31. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 50
  32. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 39
  33. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  34. 2025A Survey on LLM Mid-Trainingcited by 21
  35. 2023Tool-Augmented Reward Modelingcited by 20
  36. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulationcited by 20
  37. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 18
  38. 2025Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspectivecited by 14

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Controlcited by 3
  2. 2026Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RLcited by 1
  3. 2026Adapting Technical-Service LLM Agents with Latent Logic Augmentation, Robust Noise Reduction, and Hybrid Reward Modelingcited by 0
  4. 2026Toolcompass: Guiding Tool Trialing, Not Suppressing Itcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agents
  2. 2026Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
  3. 2025Don't Just Fine-tune the Agent, Tune the Environment
  4. 2026Learning Generalizable Behaviors for Terminal Agents
  5. 2026Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization
  6. 2025Debunk the Myth of SFT Generalization
  7. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
  8. 2025How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns