# Rethinking the Potential of Layer Freezing for Efficient DNN Training

arXiv: https://arxiv.org/abs/2508.15033 · HTML: https://mlsanity.com/paper/2508.15033/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 20 Aug 2025.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning](https://mlsanity.com/paper/1911.03090/) (2019)
- [ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training](https://mlsanity.com/paper/2104.14129/) (2021)
- [SuperMix: Supervising the Mixing Data Augmentation](https://mlsanity.com/paper/2003.05034/) (2020)
- [A Review of Deep Transfer Learning and Recent Advancements](https://mlsanity.com/paper/2201.09679/) (2022)
- [FreezeOut: Accelerate Training by Progressively Freezing Layers](https://mlsanity.com/paper/1706.04983/) (2017)
- [AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning](https://mlsanity.com/paper/2102.01386/) (2021)
- [GACT: Activation Compressed Training for Generic Network Architectures](https://mlsanity.com/paper/2206.11357/) (2022)
- [Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezing](https://mlsanity.com/paper/2201.06227/) (2022)
- [SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing](https://mlsanity.com/paper/2401.16720/) (2024)
- [Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training](https://mlsanity.com/paper/2209.11204/) (2022)
- [LoRA-C: Parameter-Efficient Fine-Tuning of Robust CNN for IoT Devices](https://mlsanity.com/paper/2410.16954/) (2024)
- [CNN Feature Map Augmentation for Single-Source Domain Generalization](https://mlsanity.com/paper/2305.16746/) (2023)
- [Data Augmentation using Feature Generation for Volumetric Medical Images](https://mlsanity.com/paper/2209.14097/) (2022)

## Abstract (arXiv)

With the growing size of deep neural networks and datasets, the computational costs of training have significantly increased. The layer-freezing technique has recently attracted great attention as a promising method to effectively reduce the cost of network training. However, in traditional layer-freezing methods, frozen layers are still required for forward propagation to generate feature maps for unfrozen layers, limiting the reduction of computation costs. To overcome this, prior works proposed a hypothetical solution, which caches feature maps from frozen layers as a new dataset, allowing later layers to train directly on stored feature maps. While this approach appears to be straightforward, it presents several major challenges that are severely overlooked by prior literature, such as how to effectively apply augmentations to feature maps and the substantial storage overhead introduced. If these overlooked challenges are not addressed, the performance of the caching method will be severely impacted and even make it infeasible. This paper is the first to comprehensively explore these challenges and provides a systematic solution. To improve training accuracy, we propose textitsimilarity-aware channel augmentation, which caches channels with high augmentation sensitivity with a minimum additional storage cost. To mitigate storage overhead, we incorporate lossy data compression into layer freezing and design a textitprogressive compression strategy, which increases compression rates as more layers are frozen, effectively reducing storage costs. Finally, our solution achieves significant reductions in training cost while maintaining model accuracy, with a minor time overhead. Additionally, we conduct a comprehensive evaluation of freezing and compression strategies, providing insights into optimizing their application for efficient DNN training.