mlsanityData to 14 Oct 2026

Three Roles, One Model: Role Orchestration at Inference Time to Close the Performance Gap Between Small and Large Agents

arXiv 2604.11465 ↗First posted 13 Apr 2026cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language model (LLM) agents show promise on realistic tool-use tasks, but deploying capable agents on modest hardware remains challenging. We study whether inference-time scaffolding alone, without any additional training compute, can improve the performance of a small model in complex multi-step environments. Operating on a single 24GB GPU, we evaluate Qwen3-8B on the AppWorld benchmark under both full-precision and 4-bit quantized configurations. Without any intervention, the raw model achieves just 5.4% (FP16) and 3.0% (AWQ) task goal completion. Guided by a systematic failure mode analysis, we introduce a three-tier inference scaffolding pipeline that deploys the same frozen model in three distinct roles: (1) a summarization model that preserves critical artifacts (tokens, credentials, API responses) while compressing dialogue history; (2) the main agent model that reasons over the compressed context; and (3) an isolated correction model that reviews and revises the agent's code output without access to conversation history, breaking repetitive failure loops. Applied to the same unmodified model, this scaffolding yields 8.9% (FP16) and 5.9% (AWQ) task goal completion, roughly doubling performance in both settings, with particularly strong gains on difficulty-1 tasks (15.8% to 26.3% FP16; 5.3% to 14.0% AWQ). On full-precision inference, our scaffolded 8B model surpasses DeepSeek-Coder 33B Instruct (7.1%) from the original AppWorld evaluation, demonstrating that structured inference-time interventions can make small models competitive with systems 4 times their size. We formalize the approach as a scaffolded policy over a frozen base model, three invocations of the same weights with different conditioning, drawing connections to test-time compute scaling and action-space shaping in reinforcement learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 29 of its 34 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

29 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025AgentEvolver: Towards Efficient Self-Evolving Agent System13 Nov
  2. 2023FireAct: Toward Language Agent Fine-tuning9 Oct

Other references · 27

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  11. 2023Teaching Large Language Models to Self-Debugcited by 988
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
15 more
  1. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  2. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  3. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  4. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  6. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 163
  7. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 127
  8. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 108
  9. 2025Recursive Language Modelscited by 104
  10. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 100
  11. 2024Tuning Language Models by Proxycited by 89
  12. 2025Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolutioncited by 73
  13. 2025Meta-RL Induces Exploration in Language Agentscited by 21
  14. 2025SPIRAL: Symbolic LLM Planning via Grounded and Reflective Searchcited by 7
  15. 2025Enhancing LLM Planning Capabilities through Intrinsic Self-Critiquecited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Harnesses for Inference-Time Alignment over Execution Trajectories
  2. 2025ACON: Optimizing Context Compression for Long-horizon LLM Agents
  3. 2026Can Small Agents Collaborate to Beat a Single Large Language Model?
  4. 2026Parallel Context Compaction for Long-Horizon LLM Agent Serving
  5. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
  6. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks
  7. 2025AgentFlux: Decoupled Fine-Tuning & Inference for On-Device Agentic Systems
  8. 2025Towards Efficient Agents: A Co-Design of Inference Architecture and System