Input-Blind Controls Produce Substantial Oracle Headroom for Layer Programs in Multiple-Choice Evaluation
Not on the map. Why?
Abstract (arXiv)
Adaptive computation aims to improve language-model inference by tailoring execution to each input. For layer programs, oracle evaluations use known answers to estimate the potential gain from this flexibility, before a practical selector is available. However, a gain from selection does not by itself explain why the chosen programs help. This study examines this distinction using 32 layer-skipping and repetition programs on two models and 4,413 multiple-choice items. The analysis compares their gains over a fixed action selected without the evaluation prompt with those of input-blind perturbations at the same sites, re-evaluating selections on another prompt. With shared option order, the controls give 10.2-11.8 and 15.6-19.4 percentage points of headroom on Qwen3-4B-Base and Llama-3.1-8B, exceeding the real programs' 9.0 and 10.1 in all three random-direction draws per model. They match answer-change rate only, and the ordering depends on the menu: in post hoc comparisons, real programs lead on Llama's repeat-only menu in every draw. A smaller KL-calibrated comparison, including an input-dependent control, favours real programs in point estimate, with inconclusive corrected tests. Fixed letter offsets produce headroom of similar scale. Rotating options sharply reduces both families' headroom, while leaving positive real-minus-control differences of 1.4-2.3 and 3.7-4.5 points; their magnitudes and statistical support depend on further adjustments and the reference. A supplementary generated-answer test finds that search-selected programs keep a 26.0-point advantage over programs selected for other problems after rewording, without a placebo comparison. These results show that substantial headroom can persist across prompts with shared option order without establishing a benefit specific to the selected layer computation; neither ordering against these controls identifies that benefit.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 47 references that are in the map. First posted on arXiv 7 Oct 2026.
Builds on
18 of 47 references.
References · 18
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2025Qwen3 Technical Reportcited by 8,195
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
- 2026Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weightscited by 25
- 2025Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMscited by 14
- 2026Training-Free Looped Transformerscited by 12
- 2025Dr.LLM: Dynamic Layer Routing in LLMscited by 10
- 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMscited by 6
- 2026What Does a Routing Oracle Measure Under Stochastic Decoding? Coupling, Scorer Choice, and Single-Commit Ceilingscited by 3
- 2026Opportunity Is Not Realizability: Selection-Valid Diagnostics for Multi-LLM Routingcited by 3
6 more
- 2026Deployable Per-Instance Multi-Layer Activation Steering for Large Language Modelscited by 2
- 2026Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gainscited by 1
- 2026MACRO: Markov Chain Routing of Transformer Layerscited by 1
- 2026Programs-of-Layers in LLMs through the Lens of Cortical Areascited by 1
- 2026More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnessescited by 1
- 2026Architectural Sampling: Test-Time Scaling via Computational Diversity in Frozen Vision-Language Modelscited by 1
Built on by
Not cited yet.