mlsanityData to 14 Oct 2026

Learning Perturbation Robust Policies for LLM Agents with Stable Optimization

arXiv 2609.34064 ↗First posted 28 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning (RL) has become an effective post-training paradigm for long-horizon large language model (LLM) agents. However, we find that the resulting policies can be sensitive to various policy perturbations, such as hidden-state noise, pruning, and quantization. In this work, we study how to improve perturbation robustness during policy optimization. We first introduce the notion of a perturbation robust policy and analyze conditions under which perturbed policy updates preserve stable monotonic improvement. Based on this analysis, we introduce Stable Perturbation-Robust Policy Optimization (SPrPO), which applies adaptive and sensitivity-aware perturbations during RL training. We evaluate SPrPO on ALFWorld and WebShop and conduct systematic experiments across multiple perturbation types and scales, showing improved perturbation robustness while maintaining stable policy optimization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 38 of its 45 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

38 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agents26 Jan
  2. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  3. 2024AGILE: A Novel Reinforcement Learning Framework of LLM Agents23 May

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  11. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  12. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
23 more
  1. 2019Importance Estimation for Neural Network Pruningcited by 541
  2. 2021A White Paper on Neural Network Quantizationcited by 472
  3. 2017Robust Adversarial Reinforcement Learningcited by 451
  4. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  5. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
  6. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  7. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  8. 2017Parameter Space Noise for Explorationcited by 161
  9. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  10. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 100
  11. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  12. 2021Policy Smoothing for Provably Robust Reinforcement Learningcited by 40
  13. 2018Wasserstein Distributionally Robust Kalman Filteringcited by 17
  14. 2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 16
  15. 2023PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizercited by 15
  16. 2021Chebyshev-Cantelli PAC-Bayes-Bennett Inequality for the Weighted Majority Votecited by 13
  17. 2022Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environmentscited by 9
  18. 2023Distributionally Robust Linear Quadratic Controlcited by 8
  19. 2024Improving robustness to corruptions with multiplicative weight perturbationscited by 7
  20. 2024Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantizationcited by 4
  21. 2025Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Modelscited by 3
  22. 2026SHAPO: Sharpness-Aware Policy Optimization for Safe Explorationcited by 2
  23. 2026Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization
  2. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  3. 2026Rethinking the Trust Region in LLM Reinforcement Learning
  4. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization
  5. 2026Stabilizing Policy Optimization via Logits Convexity
  6. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training
  7. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment
  8. 2026Towards Robust Reinforcement Learning for Small-Scale Language Model Agents