# More Is Not Always Better: Cross-Component Interference in LLM Agent Scaffolding

arXiv: https://arxiv.org/abs/2605.05716 · HTML: https://mlsanity.com/paper/2605.05716/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 22 of 22)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Efficiently Identifying Task Groupings for Multi-Task Learning](https://mlsanity.com/paper/2109.04617/) (2021)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs](https://mlsanity.com/paper/2505.11423/) (2025)
- [SHAP-IQ: Unified Approximation of any-order Shapley Interactions](https://mlsanity.com/paper/2303.01179/) (2023)
- [How Many Instructions Can LLMs Follow at Once?](https://mlsanity.com/paper/2507.11538/) (2025)
- [HarsanyiNet: Computing Accurate Shapley Values in a Single Forward Propagation](https://mlsanity.com/paper/2304.01811/) (2023)
- [ThriftLLM: On Cost-Effective Selection of Large Language Models for Classification Queries](https://mlsanity.com/paper/2501.04901/) (2025)
- [The Unreasonable Effectiveness of Eccentric Automatic Prompts](https://mlsanity.com/paper/2402.10949/) (2024)
- [Explaining Large Language Models Decisions Using Shapley Values](https://mlsanity.com/paper/2404.01332/) (2024)
- [Position Engineering: Boosting Large Language Models through Positional Information Manipulation](https://mlsanity.com/paper/2404.11216/) (2024)
- [DemoShapley: Valuation of Demonstrations for In-Context Learning](https://mlsanity.com/paper/2410.07523/) (2024)
- [On the Paradoxical Interference between Instruction-Following and Task Solving](https://mlsanity.com/paper/2601.22047/) (2026)
- [A Regression Framework for Understanding Prompt Component Impact on LLM Performance](https://mlsanity.com/paper/2603.26830/) (2026)

## Built on by (most cited first, 1 of 1)

- [How Should a Prompt Optimizer Spend a Tight Budget? BudgetAPO with Noise-Adaptive Evaluation](https://mlsanity.com/paper/2610.05671/) (2026)

## Abstract (arXiv)

LLM agent systems are built by stacking scaffolding components (planning, tools, memory, self-reflection, retrieval) assuming more is better. We study cross-component interference (CCI): degradation when components interact destructively. We run a full factorial experiment over all 2^5=32 subsets of five components on HotpotQA and GSM8K with Llama-3.1-8B/70B (96 conditions, up to 10 seeds). The All-In system is consistently suboptimal: on HotpotQA, a single-tool agent surpasses All-In by 32% (F1 0.233 vs 0.177, p=0.023); on GSM8K, a 3-component subset beats All-In by 79% (0.43 vs 0.24, p=0.010). Optimal component count is task-dependent (k*=1-4) and scale-sensitive: at 70B, combinations that hurt at 8B provide gains, though All-In still trails the best subset. We fit a main-effects regression (R^2=0.916, adj-R^2=0.899, LOOCV=0.872), compute exact Shapley values, and find 183/325 submodularity violations (56.3%), showing greedy selection is unreliable. A three-body synergy among Tool Use, Self-Reflection, and Retrieval (INT_3=+0.175, 95% CI [+0.003,+0.351]) is reported as exploratory. CCI replicates across model families (Qwen2.5) and is robust to prompt paraphrasing. Our findings suggest maximally-equipped agent defaults should be replaced by task-specific subset selection via interaction-aware analysis.