mlsanityData to 7 Oct 2026

The Expressive Power of Tuning Only the Normalization Layers

arXiv 2302.07937 ↗First posted 15 Feb 2023cs.LG cs.AI stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Feature normalization transforms such as Batch and Layer-Normalization have become indispensable ingredients of state-of-the-art deep neural networks. Recent studies on fine-tuning large pretrained models indicate that just tuning the parameters of these affine transforms can achieve high accuracy for downstream tasks. These findings open the questions about the expressive power of tuning the normalization layers of frozen networks. In this work, we take the first step towards this question and show that for random ReLU networks, fine-tuning only its normalization layers can reconstruct any target network that is O(√width) times smaller. We show that this holds even for randomly sparsified networks, under sufficient overparameterization, in agreement with prior empirical work.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 15 Feb 2023.

Builds on

35 of 48 references.

References · 35

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2016Layer Normalizationcited by 5,579
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  6. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  7. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  8. 2014How transferable are features in deep neural networks?cited by 1,216
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  10. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  11. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  12. 2018How Does Batch Normalization Help Optimization?cited by 355
23 more
  1. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  2. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  3. 2018Understanding Batch Normalizationcited by 157
  4. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 156
  5. 2019Linearized two-layers neural networks in high dimensioncited by 155
  6. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  7. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  8. 2018On the importance of single directions for generalizationcited by 99
  9. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  10. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  11. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  12. 2019A Mean Field Theory of Batch Normalizationcited by 58
  13. 2018Towards Understanding Regularization in Batch Normalizationcited by 56
  14. 2019Dense Classification and Implanting for Few-Shot Learningcited by 56
  15. 2019Approximation spaces of deep neural networkscited by 51
  16. 2017Universal representations:The missing link between faces, text, planktons, and cat breedscited by 45
  17. 2018K for the Price of 1: Parameter-efficient Multi-task and Transfer Learningcited by 34
  18. 2019Neural tangent kernels, transportation mappings, and universal approximationcited by 30
  19. 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
  20. 2021On the Approximation Power of Two-Layer Networks of Random ReLUscited by 29
  21. 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
  22. 2017Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classificationcited by 18
  23. 2022Batch Normalization Explainedcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 4; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  2. 2025Deepfake Detection that Generalizes Across Benchmarkscited by 20
  3. 2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
  4. 2024Parameter-Efficient Fine-Tuning of State Space Modelscited by 11
  5. 2025Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classificationcited by 3
  6. 2026DroneFINE: Domain-Aware Parameter-Efficient Fine-Tuning of Vision-Language Detectors for Drone Imagescited by 3
  7. 2024Expressivity of Neural Networks with Random Weights and Learned Biasescited by 2
  8. 2024Evolutionary Retrofittingcited by 2
  9. 2024Cyclic Sparse Training: Is it Enough?cited by 1
  10. 2026Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networkscited by 1
  11. 2025The impact of allocation strategies in subset learning on the expressive power of neural networkscited by 0
  12. 2025The Quest for Winning Tickets in Low-Rank Adapterscited by 0
1 more
  1. 2026Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICLcited by 0