Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation
Not on the map. Why?
Abstract (arXiv)
Production prompts rarely carry a single instruction. One system message may require valid JSON, a word limit, three citations, and a fixed tone at the same time. We study how instruction-following degrades as such constraints accumulate. We introduce a benchmark that stacks 24 verifier-checked instructions, one to twenty at a time, and evaluate three production-tier LLMs (Claude Sonnet 4.6, GPT-5-mini, Gemini 2.5 Flash). Instruction-following degrades non-linearly: the follow rate falls from ~96% to as low as 20%, driven by a structured and reproducible set of pairwise conflicts. A single "output JSON" constraint, for example, is jointly unsatisfiable with nine others. We then evaluate a training-free remedy: an instruction compiler that rewrites the stacked prompt in a single LLM call and is reused across queries. Its benefit is capability-graded. It recovers up to +11 points of follow rate for weaker models, which are also the models most often deployed at scale, while leaving stronger models, which already internalise the same structure, essentially unchanged. Cluster-robust tests, same-baseline controls, and a within-family scaling ladder attribute the gain to the rewrite itself rather than to additional tokens, reordering, or measurement headroom. We release the benchmark, verifiers, and cached runs for full reproduction.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 7 of its 12 references that are in the map. First posted on arXiv 31 Jul 2026.
Builds on
7 of 12 references.
References · 7
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
- 2024PRewrite: Prompt Rewriting with Reinforcement Learningcited by 43
- 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 34
- 2025RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Datacited by 11
- 2026Brevity Constraints Reverse Performance Hierarchies in Language Modelscited by 4
- 2026On the Paradoxical Interference between Instruction-Following and Task Solvingcited by 2
Built on by
Not cited yet.