Dynamic Parameterization Is Not Dynamic Inference
Not on the map. Why?
Abstract (arXiv)
Input-dependent controller coefficients are often treated as evidence of dynamic inference or computational savings. This interpretation conflates three properties: coefficient variation, dependence of a frozen model on how coefficients are assigned to inputs, and conditional execution. We focus on the second property and formulate a general principle of frozen-controller auditing. We provide one concrete implementation, Frozen-Controller Auditing (FCA), which caches the complete coefficient tensor along an unperturbed trajectory, disables the controller, and replays the frozen model with cross-input reassignment, token shuffling, and static profiles estimated from an independent calibration set. Because the coefficients are cached before any intervention, performance changes under replay measure assignment dependence without feedback from recomputing the controller on perturbed hidden states. Across seven independently trained 76M FeatureGate Transformers and three 504M models, static layerwise profiles retain 98.70% and 99.43% of the Correct-to-GlobalMean performance gap, respectively. Layer identity explains 87% to 96% of the coefficient variance. FeatureGate nevertheless executes every Transformer block, and its measured inference is 30.8% slower than Dense. On the public MUDDPythia-1.4B checkpoint, cross-input reassignment and token shuffling increase NLL by 1.9067 and 2.9637, respectively. These penalties show that the model depends strongly on content-conditioned cross-layer assignment. MUDDPythia also executes every Transformer block. The results show that dynamic parameterization alone does not establish dynamic inference and that functional dynamics do not establish computational savings. Claims about dynamic models should separately report coefficient variation, functional dependence of the frozen model, and actual execution.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 28 Jul 2026.
Builds on
43 of 53 references.
References · 43
- 2017Attention Is All You Needcited by 59,932
- 2016Densely Connected Convolutional Networkscited by 6,691
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
- 2016HyperNetworkscited by 1,003
- 2022Token Merging: Your ViT But Fastercited by 995
- 2016Deep Networks with Stochastic Depthcited by 973
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
31 more
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2021Going deeper with Image Transformerscited by 759
- 2021Scaling Vision with Sparse Mixture of Expertscited by 740
- 2018Universal Transformerscited by 613
- 2022Mixture-of-Experts with Expert Choice Routingcited by 611
- 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
- 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
- 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
- 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
- 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
- 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
- 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
- 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
- 2016Dynamic Filter Networkscited by 318
- 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
- 2019Learning Deep Transformer Models for Machine Translationcited by 309
- 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
- 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
- 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
- 2019Depth-Adaptive Transformercited by 227
- 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
- 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
- 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
- 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
- 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
- 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
- 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
- 2020RealFormer: Transformer Likes Residual Attentioncited by 62
- 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 48
- 2018Training Deeper Neural Machine Translation Models with Transparent Attentioncited by 37
- 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
Built on by
Not cited yet.