Model of Models: When Does Emitting a Specialist Beat Attending, Adapting, or Tuning?
Not on the map. Why?
Abstract (arXiv)
Given a task described by a few examples, how should a model be specialized to it? Four mechanisms are available -- zero-shot, in-context attention, test-time gradient adaptation, and emitting specialist weights from a hypernetwork -- yet the operating regime of the last is rarely mapped. We run the identical four-way comparison across six tasks spanning regression, generation, language modeling, reinforcement learning, and clinical and genomic classification, holding the specialist, the context, and (where we can) the training budget fixed. The clearest wins for emission are about cost at matched quality: it ties the state-of-the-art amortized tabular model (TabPFN) on clinical few-shot classification while emitting a reusable specialist instead of re-attending the support set per query, and reaches noise-floor shape generation with a 132-float per-instance program. On few-shot sinusoid regression it is 2--3 orders of magnitude below MAML at zero test-time gradient steps -- a margin that narrows to ~30× but persists once training budgets are equalized. Emission cannot match in-context attention on high-dimensional sequence modeling: under matched-budget pre-training a one-pass adapter recovers only a minority of the in-context gain (14.0±0.9% at 5M, 11.2±0.5% at 15M), and a LoRA-rank sweep shows this shortfall is a partial capacity limit -- capture climbs from 5% to 21% as rank grows but plateaus far below full recovery. Mechanism ablations confirm the emitted specialist is genuinely task-conditioned, not a memorized prior; and, more speculatively, emitted specialists compose in weight space -- interpolating two of them tracks the corresponding blend of their functions. We close with a falsifiable thesis, operationalized through a per-task resolution measure, bounding when each conditioning mechanism should be preferred.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 20 Jul 2026.
Builds on
14 of 18 references.
References · 14
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
- 2022Editing Models with Task Arithmeticcited by 1,120
- 2016HyperNetworkscited by 1,027
- 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
- 2018Conditional Neural Processescited by 426
- 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
- 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 137
- 2019Training Neural Networks with Local Error Signalscited by 136
- 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 115
- 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 46
- 2022Hypernetworks in Meta-Reinforcement Learningcited by 42
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Picking the Right Specialist: Attentive Neural Process-based Selection of Task-Specialized Models as Tools for Agentic Healthcare Systems
- 2023Specialist or Generalist? Instruction Tuning for Specific NLP Tasks
- 2026Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMs
- 2025Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models
- 2025Can Generalist Vision Language Models (VLMs) Rival Specialist Medical VLMs? Benchmarking and Strategic Insights
- 2024GSCo: Towards Generalizable AI in Medicine via Generalist-Specialist Collaboration
- 2025Foundation Models in Biomedical Imaging: Turning Hype into Reality
- 2026Why Specialist Models Still Matter: A Heterogeneous Multi-Agent Paradigm for Medical Artificial Intelligence