mlsanityData to 7 Oct 2026

Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting

arXiv 2410.19920 ↗First posted 25 Oct 2024cs.LGcited by 4

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) is a promising approach for aligning large language models (LLMs) knowledge with sequential decision-making tasks. However, few studies have thoroughly investigated the impact on LLM agents capabilities of fine-tuning them with RL in a specific environment. In this paper, we propose a novel framework to analyze the sensitivity of LLMs to prompt formulations following RL training in a textual environment. Our findings reveal that the performance of LLMs degrades when faced with prompt formulations different from those used during the RL training phase. Besides, we analyze the source of this sensitivity by examining the model's internal representations and salient tokens. Finally, we propose to use a contrastive loss to mitigate this sensitivity and improve the robustness and generalization capabilities of LLMs.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 29 of its 48 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

29 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb
  2. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models10 Nov
  3. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization4 Aug

Other references · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  10. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  11. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  12. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
14 more
  1. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  2. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  3. 2023Large Language Models for Robotics: A Surveycited by 132
  4. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  5. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 97
  6. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  7. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  8. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  9. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  10. 2020Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselinescited by 37
  11. 2024True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learningcited by 36
  12. 2023Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameterscited by 32
  13. 2023Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuningcited by 15
  14. 2023Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Procedural Memory Is Not All You Need: Bridging Cognitive Gaps in LLM-Based Agentscited by 9
  2. 2026SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewardscited by 9
  3. 2026Prompt-Robust Language Models: Which Training Strategies Work?cited by 0
  4. 2026WEFT: Scaling Tool-Use Post-Training for General-Purpose Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  2. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  3. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
  4. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning
  5. 2025Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learning
  6. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  7. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
  8. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier