mlsanityData to 7 Oct 2026

FreezeOut: Accelerate Training by Progressively Freezing Layers

arXiv 1706.04983 ↗First posted 15 Jun 2017stat.ML cs.LGcited by 54

Not on the map. Why?

Abstract (arXiv)

The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. Through experiments on CIFAR, we empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets, a 20% speedup without loss of accuracy for ResNets, and no improvement for VGG networks. Our code is publicly available at https://github.com/ajbrock/FreezeOut

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 8; 2023: 9; 2024: 9; 2025: 12; 2026: 7.

54 papers.

Sort:

By area:

  1. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  2. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  3. 2022Towards Adversarial Evaluations for Inexact Machine Unlearningcited by 82
  4. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  5. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  6. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  7. 2021Layer-wise Adaptive Model Aggregation for Scalable Federated Learningcited by 35
  8. 2021Learning where to learn: Gradient sparsity in meta and continual learningcited by 33
  9. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  10. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  11. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  12. 2021Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddingscited by 12
38 more
  1. 2024SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezingcited by 12
  2. 2022Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Trainingcited by 11
  3. 2024Outlier-weighed Layerwise Sampling for LLM Fine-tuningcited by 8
  4. 2024Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Modelscited by 8
  5. 2021Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezingcited by 7
  6. 2022GlueFL: Reconciling Client Sampling and Model Masking for Bandwidth Efficient Federated Learningcited by 7
  7. 2024Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Samplingcited by 7
  8. 2022PST: Plant segmentation transformer for 3D point clouds of rapeseed plants at the podding stagecited by 6
  9. 2022Aergia: Leveraging Heterogeneity in Federated Learning Systemscited by 6
  10. 2023Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exitingcited by 6
  11. 2022Embedding Recycling for Language Modelscited by 5
  12. 2023Efficient Self-supervised Continual Learning with Progressive Task-correlated Layer Freezingcited by 5
  13. 2025PFedDST: Personalized Federated Learning with Decentralized Selection Trainingcited by 4
  14. 2025Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Datacited by 4
  15. 2025LayerLock: Non-collapsing Representation Learning with Progressive Freezingcited by 3
  16. 2021Self-Supervised Monocular Depth Estimation of Untextured Indoor Rotated Scenescited by 2
  17. 2021Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusterscited by 2
  18. 2023Benchmarking Deep Learning Frameworks for Automated Diagnosis of Ocular Toxoplasmosis: A Comprehensive Approach to Classification and Segmentationcited by 2
  19. 2025Frozen Layers: Memory-efficient Many-fidelity Hyperparameter Optimizationcited by 2
  20. 2021Weighted Neural Tangent Kernel: A Generalized and Improved Network-Induced Kernelcited by 1
  21. 2023Text recognition on images using pre-trained CNNcited by 1
  22. 2023Weight Compander: A Simple Weight Reparameterization for Regularizationcited by 1
  23. 2023Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learningcited by 1
  24. 2024NeuroFlux: Memory-Efficient CNN Training Using Adaptive Local Learningcited by 1
  25. 2025AdaGradSelect: An adaptive gradient-guided layer selection method for efficient fine-tuning of SLMscited by 1
  26. 2026WSBD: Freezing-Based Optimizer for Quantum Neural Networkscited by 1
  27. 2021Attempted Blind Constrained Descent Experimentscited by 0
  28. 2023Toward efficient resource utilization at edge nodes in federated learningcited by 0
  29. 2024Hybrid Deep Learning-Based for Enhanced Occlusion Segmentation in PICU Patient Monitoringcited by 0
  30. 2025ICE-Pruning: An Iterative Cost-Efficient Pruning Pipeline for Deep Neural Networkscited by 0
  31. 2025Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networkscited by 0
  32. 2025Rethinking the Potential of Layer Freezing for Efficient DNN Trainingcited by 0
  33. 2025AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learningcited by 0
  34. 2025HyCoVAD: A Hybrid SSL-LLM Model for Complex Video Anomaly Detectioncited by 0
  35. 2025GLAI: GreenLightningAI for Accelerated Training through Knowledge Decouplingcited by 0
  36. 2025Beyond Size and Growth: Rethinking Lung Cancer Screening with AI Based Nodule Detection and Diagnosiscited by 0
  37. 2026EPAS: Efficient Training with Progressive Activation Sharingcited by 0
  38. 2026TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelismcited by 0

50 most cited of 54 All 54 (JSON)