# Steer, Don't Solve: Training Small Critic Models for Large Code Agents

arXiv: https://arxiv.org/abs/2606.21811 · HTML: https://mlsanity.com/paper/2606.21811/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 20 Jun 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 20 of 20)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2503.09572/) (2025)
- [R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents](https://mlsanity.com/paper/2504.07164/) (2025)
- [LLM Critics Help Catch LLM Bugs](https://mlsanity.com/paper/2407.00215/) (2024)
- [SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents](https://mlsanity.com/paper/2601.16746/) (2026)
- [LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback](https://mlsanity.com/paper/2406.14024/) (2024)
- [When Agents go Astray: Course-Correcting SWE Agents with PRMs](https://mlsanity.com/paper/2509.02360/) (2025)
- [How to Train Your Advisor: Steering Black-Box LLMs with Advisor Models](https://mlsanity.com/paper/2510.02453/) (2025)
- [An Empirical Study on Failures in Automated Issue Solving](https://mlsanity.com/paper/2509.13941/) (2025)
- [Improving Model Factuality with Fine-grained Critique-based Evaluator](https://mlsanity.com/paper/2410.18359/) (2024)
- [Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development](https://mlsanity.com/paper/2505.16086/) (2025)
- [ProRefine: Inference-Time Prompt Refinement with Textual Feedback](https://mlsanity.com/paper/2506.05305/) (2025)
- [Large Language Model Critics for Execution-Free Evaluation of Code Changes](https://mlsanity.com/paper/2501.16655/) (2025)
- [Hybrid-Gym: Training Coding Agents to Generalize Across Tasks](https://mlsanity.com/paper/2602.16819/) (2026)

## Built on by (most cited first, 5 of 5)

- [Learning When and How to Intervene: A Hindsight-Distilled Sentinel for Coding Agents](https://mlsanity.com/paper/2609.39957/) (2026)
- [What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents](https://mlsanity.com/paper/2609.39989/) (2026)
- [Training Advisors for LLM Agents from Task Outcomes](https://mlsanity.com/paper/2610.09858/) (2026)
- [SWE-Critix: Post-Training Generalist Verifiers for Assessing Coding-Agent Trajectories](https://mlsanity.com/paper/2610.13356/) (2026)
- [JevCoder: When Does Decision Delegation Pay Off in Code Repair?](https://mlsanity.com/paper/2610.19596/) (2026)

## Abstract (arXiv)

Coding tasks are typically complicated and require multiple capabilities, ranging from high-level planning to low-level implementation. While coding agents are optimized for the joint capabilities, individual capabilities such as high-level planning may have different optima and remain a major bottleneck. To address this challenge, we train a separate critic model that is specialized in high-level planning to steer the coding agent in inference. We construct SFT and DPO data to train the critic model to identify errors made by the coding agent and provide correct and clear high-level guidance without generating concrete actions. Experiments show that our fine-tuned 4B and 8B critic models significantly improve the performance of 6 larger coding agents (e.g., improving the resolved rates of GLM-4.7-Flash-30B-A3B and GPT-OSS-120B by 16.0% and 14.4% on SWE-Bench Verified). The critic model also reduces the total inference costs for some coding agents by solving tasks in fewer steps (e.g., reducing the per-example inference cost for GPT-OSS-20B from 0.07 to 0.03). Code: https://github.com/shubhamrgandhi/critic-training