mlsanityData to 7 Oct 2026

Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models

arXiv 2409.09345 ↗First posted 14 Sep 2024cs.AIcited by 28

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Agents significantly enhance the capabilities of standalone Large Language Models (LLMs) by perceiving environments, making decisions, and executing actions. However, LLM agents still face challenges in tasks that require multiple decision-making steps. Estimating the value of actions in specific tasks is difficult when intermediate actions are neither appropriately rewarded nor penalized. In this paper, we propose leveraging a task-relevant Q-value model to guide action selection. Specifically, we first collect decision-making trajectories annotated with step-level Q values via Monte Carlo Tree Search (MCTS) and construct preference data. We then use another LLM to fit these preferences through step-level Direct Policy Optimization (DPO), which serves as the Q-value model. During inference, at each decision-making step, LLM agents select the action with the highest Q value before interacting with the environment. We apply our method to various open-source and API-based LLM agents, demonstrating that Q-value models significantly improve their performance. Notably, the performance of the agent built with Phi-3-mini-4k-instruct improved by 103% on WebShop and 75% on HotPotQA when enhanced with Q-value models, even surpassing GPT-4o-mini. Additionally, Q-value models offer several advantages, such as generalization to different LLM agents and seamless integration with existing prompting strategies.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 34 references that are in the map. First posted on arXiv 14 Sep 2024.

Builds on

23 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents4 Mar
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 21

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  9. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  10. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  11. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  12. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
9 more
  1. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  2. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  4. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  5. 2024Tree Search for Language Model Agentscited by 148
  6. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  7. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  8. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 63
  9. 2023Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoningcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 18; 2026: 9.

28 papers.

Sort:

By area:

  1. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  2. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  3. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progresscited by 69
  4. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  5. 2025AgentRM: Enhancing Agent Generalization with Reward Modelingcited by 25
  6. 2024Aviary: training language agents on challenging scientific taskscited by 23
  7. 2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Searchcited by 20
  8. 2025Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agentscited by 19
  9. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  10. 2025SAND: Boosting LLM Agents with Self-Taught Action Deliberationcited by 14
  11. 2025Exploring the Necessity of Reasoning in LLM-based Agent Scenarioscited by 10
  12. 2025Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmarkcited by 9
16 more
  1. 2025Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learningcited by 9
  2. 2026On Protecting Agentic Systems' Intellectual Property via Watermarkingcited by 7
  3. 2025DSADF: Thinking Fast and Slow for Decision Makingcited by 6
  4. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 6
  5. 2025ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generationcited by 5
  6. 2025Surfer 2: The Next Generation of Cross-Platform Computer Use Agentscited by 5
  7. 2026From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skillscited by 5
  8. 2025W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Searchcited by 4
  9. 2026Can RL Improve Generalization of LLM Agents? An Empirical Studycited by 4
  10. 2025FinPos: A Position-Aware Trading Agent System for Real Financial Marketscited by 3
  11. 2026From Pixels to Policies: Reinforcing Spatial Reasoning in Language Models for Content-Aware Layout Designcited by 1
  12. 2026Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agentscited by 1
  13. 2026GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflowscited by 1
  14. 2025A Training-free LLM Framework with Interaction between Contextually Related Subtasks in Solving Complex Taskscited by 0
  15. 2026Latent Action Reparameterization for Efficient Agent Inferencecited by 0
  16. 2026GUI agent: Guided Exploration of User-Sensitive Screenscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planning
  2. 2025A Survey on the Optimization of Large Language Model-based Agents
  3. 2023Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach
  4. 2024From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning
  5. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
  6. 2026Verified Critical Step Optimization for LLM Agents
  7. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities
  8. 2024STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Making