# Scaffold Effects on GAIA: A Controlled Comparison

arXiv: https://arxiv.org/abs/2606.08529 · HTML: https://mlsanity.com/paper/2606.08529/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 13 references that are in the map.
- First posted on arXiv 7 Jun 2026.

## Builds on (most cited first, 8 of 8)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety](https://mlsanity.com/paper/2603.10044/) (2026)
- [Forecasting Frontier Language Model Agent Capabilities](https://mlsanity.com/paper/2502.15850/) (2025)
- [What AI evaluations for preventing catastrophic risks can and cannot do](https://mlsanity.com/paper/2412.08653/) (2024)
- [Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation](https://mlsanity.com/paper/2605.04135/) (2026)

## Built on by (most cited first, 5 of 5)

- [Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents](https://mlsanity.com/paper/2609.22259/) (2026)
- [EvoIn: Bridging Evolution and Internalization for Agent Fine-Tuning](https://mlsanity.com/paper/2609.35290/) (2026)
- [Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks](https://mlsanity.com/paper/2610.02001/) (2026)
- [What Does a Harness Buy? Tokens, Mostly](https://mlsanity.com/paper/2610.04433/) (2026)
- [MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks](https://mlsanity.com/paper/2610.05778/) (2026)

## Abstract (arXiv)

Published agent capability scores conflate what a model can do with what its scaffold lets it do, and the magnitude of this elicitation gap is not well characterized under controlled conditions. This study executes a pre-registered controlled comparison of three scaffolds (ReAct, a Planner-Actor-Rater multi-agent design, and planner-then-executor) across five models from three providers (Claude Opus 4.7, Sonnet 4.6, Haiku 4.5; Gemini 3.1 Pro Preview; GPT-5.5) on GAIA validation Levels 1 and 2, holding tasks and conditions fixed, with three attempts per question. Scaffold choice alone moves measured accuracy by as much as 28 percentage points within a single model (Opus, Level 2, robust slice), confirming the pre-registered hypothesis that scaffold variation produces gaps of at least 10 points. The pre-registered prediction that more capable models would be less scaffold-sensitive is rejected in direction: scaffold effects vary significantly by model in every dataset slice, but the most capable Anthropic model gains the most from structured scaffolds at the harder level, and tier-scaling holds only at Level 1 under the robust slice. The multi-agent advantage over ReAct at Level 2 appears within the Anthropic family but not for the cross-provider models, making model family rather than capability tier the conditioning variable, and the predicted planner-executor advantage on file-reading tasks is falsified. Structured scaffolds make fewer tool calls yet recover more often from mid-trajectory errors at the harder level, and a single cell (Gemini with planner-then-executor) is the cheapest at both levels and the most accurate at Level 2. These results indicate that single-scaffold capability numbers are scaffold-conditional estimates and that the elicitation gap is not guaranteed to shrink as models improve.