# The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement

arXiv: https://arxiv.org/abs/2503.16024 · HTML: https://mlsanity.com/paper/2503.16024/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [Training Language Models to Self-Correct via Reinforcement Learning](https://mlsanity.com/paper/2409.12917/) (2024)
- [AgentTuning: Enabling Generalized Agent Abilities for LLMs](https://mlsanity.com/paper/2310.12823/) (2023)
- [Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents](https://mlsanity.com/paper/2403.02502/) (2024)
- [ADaPT: As-Needed Decomposition and Planning with Language Models](https://mlsanity.com/paper/2311.05772/) (2023)
- [Recursive Introspection: Teaching Language Model Agents How to Self-Improve](https://mlsanity.com/paper/2407.18219/) (2024)
- [EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction](https://mlsanity.com/paper/2401.06201/) (2024)
- [The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks](https://mlsanity.com/paper/2502.08235/) (2025)
- [AgentGym: Evolving Large Language Model-based Agents across Diverse Environments](https://mlsanity.com/paper/2406.04151/) (2024)
- [Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models](https://mlsanity.com/paper/2403.12881/) (2024)
- [OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning](https://mlsanity.com/paper/2311.09724/) (2023)
- [ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent](https://mlsanity.com/paper/2312.10003/) (2023)
- [Self-Evolving Multi-Agent Collaboration Networks for Software Development](https://mlsanity.com/paper/2410.16946/) (2024)
- [Direct Multi-Turn Preference Optimization for Language Agents](https://mlsanity.com/paper/2406.14868/) (2024)
- [AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories](https://mlsanity.com/paper/2410.07706/) (2024)
- [Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills](https://mlsanity.com/paper/2402.03244/) (2024)
- [SelfGoal: Your Language Agents Already Know How to Achieve High-level Goals](https://mlsanity.com/paper/2406.04784/) (2024)
- [Reward Learning from Multiple Feedback Types](https://mlsanity.com/paper/2502.21038/) (2025)
- [SELU: Self-Learning Embodied MLLMs in Unknown Environments](https://mlsanity.com/paper/2410.03303/) (2024)

## Built on by (most cited first, 20 of 20)

- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning](https://mlsanity.com/paper/2504.19162/) (2025)
- [A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy](https://mlsanity.com/paper/2506.09420/) (2025)
- [No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning](https://mlsanity.com/paper/2601.06794/) (2026)
- [Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation](https://mlsanity.com/paper/2510.19897/) (2025)
- [LogicGuard: Improving Embodied LLM agents through Temporal Logic based Critics](https://mlsanity.com/paper/2507.03293/) (2025)
- [Multi-Agent GraphRAG: A Text-to-Cypher Framework for Labeled Property Graphs](https://mlsanity.com/paper/2511.08274/) (2025)
- [Think Fast and Slow: Step-Level Cognitive Depth Adaptation for LLM Agents](https://mlsanity.com/paper/2602.12662/) (2026)
- [Asymmetric Actor-Critic for Multi-turn LLM Agents](https://mlsanity.com/paper/2604.00304/) (2026)
- [Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction](https://mlsanity.com/paper/2504.17671/) (2025)
- [When Does Intrinsic Self-Correction Help? A Task-Sensitive Analysis](https://mlsanity.com/paper/2606.23196/) (2026)
- [User-Oriented Multi-Turn Dialogue Generation with Tool Use at scale](https://mlsanity.com/paper/2601.08225/) (2026)
- [Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments](https://mlsanity.com/paper/2606.03698/) (2026)
- [Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents](https://mlsanity.com/paper/2608.21027/) (2026)
- [LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization](https://mlsanity.com/paper/2604.25130/) (2026)
- [A History-Aware Visually Grounded Critic for Computer Use Agents](https://mlsanity.com/paper/2606.11078/) (2026)
- [Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generation](https://mlsanity.com/paper/2608.30397/) (2026)
- [medR: Reward Engineering for Clinical Offline Reinforcement Learning via Tri-Drive Potential Functions](https://mlsanity.com/paper/2602.03305/) (2026)
- [A Study of the Reliability of Agentic AI-Generated Programs](https://mlsanity.com/paper/2609.18298/) (2026)
- [Beyond Corrected Actions: Training LLM Agents on Textual Critiques](https://mlsanity.com/paper/2610.19367/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have recently transformed from text-based assistants to autonomous agents capable of planning, reasoning, and iteratively improving their actions. While numerical reward signals and verifiers can effectively rank candidate actions, they often provide limited contextual guidance. In contrast, natural language feedback better aligns with the generative capabilities of LLMs, providing richer and more actionable suggestions. However, parsing and implementing this feedback effectively can be challenging for LLM-based agents. In this work, we introduce Critique-Guided Improvement (CGI), a novel two-player framework, comprising an actor model that explores an environment and a critic model that generates detailed nature language feedback. By training the critic to produce fine-grained assessments and actionable revisions, and the actor to utilize these critiques, our approach promotes more robust exploration of alternative strategies while avoiding local optima. Experiments in three interactive environments show that CGI outperforms existing baselines by a substantial margin. Notably, even a small critic model surpasses GPT-4 in feedback quality. The resulting actor achieves state-of-the-art performance, demonstrating the power of explicit iterative guidance to enhance decision-making in LLM-based agents.