On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows

arXiv 2504.01931 ↗First posted 2 Apr 2025cs.CLcited by 4

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more

Abstract (arXiv)

Agentic AI workflows (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low. A promising solution is inference-time alignment, which uses extra compute at test time to improve performance. Inference-time alignment relies on three components: sampling, evaluation, and feedback. While most prior work studies sampling and automatic evaluation, feedback remains underexplored. To study the role of feedback, we introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques (reward models or AI-generated textual feedback) between decoding steps. Through IAD, we analyze feedback along four dimensions: (1) its role in the accuracy-compute trade-offs with limited inference budget, (2) quantifying the gains over diversity-only baselines such as best-of-N sampling, (3) effectiveness of composing feedback from reward models versus textual critique, and (4) robustness to noisy or low-quality feedback. Across Sketch2Code, Text2SQL, Intercode, and WebShop, we show that IAD with proper integration of high fidelity feedback leads to consistent gains up to 10 percent absolute performance improvement over various baselines such as best-of-N. Our findings underscore feedback as a crucial knob for inference-time alignment of agentic AI workflows with limited inference budget.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 37 of its 46 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

37 of 46 references.

Closest ancestors · 2

Same direction, newest first

2024Tool-Planner: Task Planning with Clusters across Multiple Tools6 Jun
2023Teaching Large Language Models to Self-Debug11 Apr

Other references · 35

2023GPT-4 Technical Reportcited by 18,396
2022Training language models to follow instructions with human feedbackcited by 15,971
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
2023Mistral 7Bcited by 3,164
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420

23 more

2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
2023Large Language Models as Optimizerscited by 864
2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 246
2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
2024CHESS: Contextual Harnessing for Efficient SQL Synthesiscited by 169
2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 164
2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 138
2023Controlled Decoding from Language Modelscited by 135
2023Self-planning Code Generation with Large Language Modelscited by 124
2024Theoretical guarantees on the best-of-n alignment policycited by 124
2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
2023FLIRT: Feedback Loop In-context Red Teamingcited by 83
2024AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agentscited by 74
2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 72
2024The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Modelscited by 67
2024Variational Best-of-N Alignmentcited by 51
2024A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQLcited by 46
2024Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototypingcited by 29
2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 25
2024Information Theoretic Guarantees For Policy Alignment In Large Language Modelscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

2026Structured Scaling of AI Discovery Across Diverse Scientific Domainscited by 21
2025RAISE: Reasoning Agent for Interactive SQL Explorationcited by 4
2026ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulationcited by 2
2025ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Modelscited by 1

Similar, not cited

Close in wording, no citation either way