Small Models Scout Bottleneck Order for Large-Model Data Control
Not on the map. Why?
Abstract (arXiv)
Small proxy models are commonly used to identify data mixtures for larger-scale training. We ask whether their training trajectories reveal another transferable structure: the order in which larger models should resolve skill bottlenecks. We formulate first-passage skill training, where each monitored skill has a target floor and the objective is to minimize the tokens required to reach all floors. We introduce LogFloor, a closed-loop controller that directs each round toward current bottlenecks, producing phase-ordered resolution trajectories. Across five bAbI skill slices on Qwen2.5-1.5B, LogFloor reduces token cost by 56.2% on average. In 70M-to-12B transfer, three-round replay of a 70M scout path reaches every floor in all eight target runs, saving 30.9% by pair mean, 39.4% in pooled training tokens, and 37.6% under source-cost accounting. On MMLU-control, a frozen scout path succeeds across all eight 12B runs. Collapsing a path to its static marginal mixture or reversing its phase order removes most benefits, while bottleneck labels alone remain partially useful. These results identify phase-ordered bottleneck resolution as a transferable curriculum structure for monitored skill-targeted training.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 13 references that are in the map. First posted on arXiv 14 Aug 2026.
Builds on
13 of 13 references.
References · 13
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
- 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
- 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
- 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 44
- 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 34
- 2025Data Mixing Can Induce Phase Transitions in Knowledge Acquisitioncited by 16
- 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 10
- 2025TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-trainingcited by 6
- 2026Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamicscited by 6
- 2026Dynamic Proxy-Mixing: Transferring Replay Controllers from Small to Large Models for Continual Instruction Tuningcited by 3
- 2025AC-ODM: Actor--Critic Online Data Mixing for Sample-Efficient LLM Pretrainingcited by 2
1 more
Built on by
Not cited yet.