mlsanityData to 14 Oct 2026

LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities

arXiv 2504.16078 ↗First posted 22 Apr 2025cs.LG cs.AIcited by 41

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

The success of Large Language Models (LLMs) has sparked interest in various agentic applications. A key hypothesis is that LLMs, leveraging common sense and Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently solve complex domains. However, LLM agents have been found to suffer from sub-optimal exploration and the knowing-doing gap, the inability to effectively act on knowledge present in the model. In this work, we systematically study why LLMs perform sub-optimally in decision-making scenarios. In particular, we closely examine three prevalent failure modes: greediness, frequency bias, and the knowing-doing gap. We propose mitigation of these shortcomings by fine-tuning via Reinforcement Learning (RL) on self-generated CoT rationales. Our experiments across multi-armed bandits, contextual bandits, and Tic-tac-toe, demonstrate that RL fine-tuning enhances the decision-making abilities of LLMs by increasing exploration and narrowing the knowing-doing gap. Finally, we study both classic exploration mechanisms, such as ε-greedy, and LLM-specific approaches, such as self-correction and self-consistency, to enable more effective fine-tuning of LLMs for decision-making.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 78 of its 107 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

78 of 107 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games20 Nov
  2. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 75

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
38 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  11. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  12. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  13. 2018Exploration by Random Network Distillationcited by 993
  14. 2018DeepMind Control Suitecited by 956
  15. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  16. 2022A Generalist Agentcited by 808
  17. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  18. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  19. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  20. 2019Introduction to Multi-Armed Banditscited by 481
  21. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  22. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  23. 2016Learning to reinforcement learncited by 377
  24. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  25. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  26. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  27. 2022Generating Sequences by Learning to Self-Correctcited by 277
  28. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  29. 2024Capabilities of Gemini Models in Medicinecited by 237
  30. 2022Multi-Game Decision Transformerscited by 214
  31. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  32. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  33. 2023Large Language Models as General Pattern Machinescited by 197
  34. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  35. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  36. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  37. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  38. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 23.

41 papers.

Sort:

By area:

  1. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 84
  4. 2025Kevin: Multi-Turn RL for Generating CUDA Kernelscited by 78
  5. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13
  6. 2026The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?cited by 10
  7. 2026RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Formatcited by 9
  8. 2026The Illusion of Stochasticity in LLMscited by 9
  9. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  10. 20253DReasonKnee: Advancing Grounded Reasoning in Medical Vision Language Modelscited by 6
  11. 2026MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Modelscited by 6
  12. 2025ReflecSched: Solving Dynamic Flexible Job-Shop Scheduling via LLM-Powered Hierarchical Reflectioncited by 5
29 more
  1. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 4
  2. 2025When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Trainingcited by 3
  3. 2025Report of the 2025 Workshop on Next-Generation Ecosystems for Scientific Computing: Harnessing Community, Software, and AI for Cross-Disciplinary Team Sciencecited by 2
  4. 2025Memory-Augmented State Machine Prompting: A Novel LLM Agent Framework for Real-Time Strategy Gamescited by 2
  5. 2025Large Language Models Develop Novel Social Biases Through Adaptive Explorationcited by 2
  6. 2026Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experimentcited by 2
  7. 2025BioBlue: Systematic runaway-optimiser-like LLM failure modes on biologically and economically aligned AI safety benchmarks for LLMs with simplified observation formatcited by 1
  8. 2025Alternatives To Next Token Prediction In Text Generation -- A Surveycited by 1
  9. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  10. 2026STRUCTUREDAGENT: Planning with AND/OR Trees for Long-Horizon Web Taskscited by 1
  11. 2026Counteractive RL: Rethinking Core Principles for Efficient and Scalable Deep Reinforcement Learningcited by 1
  12. 2026PriorZero: Bridging Language Priors and World Models for Decision Makingcited by 1
  13. 2026STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oraclecited by 1
  14. 2025Input-Time Scaling: Adding Noise and Irrelevance into Less-Is-More Drastically Improves Reasoning Performance and Efficiencycited by 0
  15. 2025PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMscited by 0
  16. 2025ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspectivecited by 0
  17. 2025RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedbackcited by 0
  18. 2026PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficientcited by 0
  19. 2026When Do We Need LLMs? A Diagnostic for Language-Driven Banditscited by 0
  20. 2026DORA Explorer: Improving the Exploration Ability of LLMs Without Trainingcited by 0
  21. 2026Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imagingcited by 0
  22. 2026CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VIcited by 0
  23. 2026Decodable In-Context State and Model Output Across Trainingcited by 0
  24. 2026LLM sequential decision making under uncertainty in biochemical domainscited by 0
  25. 2026DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?cited by 0
  26. 2026Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisionscited by 0
  27. 2026Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgmentcited by 0
  28. 2026The belief-action gap in language model agentscited by 0
  29. 2026Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loopscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  2. 2025Rectifying LLM Thought from Lens of Optimization
  3. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Making
  4. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  5. 2025Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
  6. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Games
  7. 2026How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws
  8. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents