mlsanityData to 13 Oct 2026

Net2Net: Accelerating Learning via Knowledge Transfer

arXiv 1511.05641 ↗First posted 18 Nov 2015cs.LGcited by 246

Not on the map. Why?

Abstract (arXiv)

We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net. During real-world workflows, one often trains very many different neural networks during the experimentation and design process. This is a wasteful process in which each new model is trained from scratch. Our Net2Net technique accelerates the experimentation process by instantaneously transferring the knowledge from a previous network to each new deeper or wider network. Our techniques are based on the concept of function-preserving transformations between neural network specifications. This differs from previous approaches to pre-training that altered the function represented by a neural net when adding layers to it. Using our knowledge transfer mechanism to add depth to Inception modules, we demonstrate a new state of the art accuracy rating on the ImageNet dataset.

Key facts. Cited by 246 papers in the map (data through 13 Oct 2026). First posted on arXiv 18 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 40; 2022: 45; 2023: 34; 2024: 43; 2025: 35; 2026: 49.

246 papers.

Sort:

By area:

  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
  2. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 358
  3. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
  4. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 208
  5. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  6. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  7. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 104
  8. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  9. 2021Practical Blind Membership Inference Attack via Differential Comparisonscited by 77
  10. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  11. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 72
  12. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
38 more
  1. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 66
  2. 2023RMM: Reinforced Memory Management for Class-Incremental Learningcited by 66
  3. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 65
  4. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 65
  5. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 64
  6. 2022Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weightscited by 63
  7. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 62
  8. 2021Distributed Deep Learning in Open Collaborationscited by 58
  9. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  10. 2025LongCat-Flash Technical Reportcited by 55
  11. 2022Staged Training for Transformer Language Modelscited by 49
  12. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 48
  13. 2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 43
  14. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 42
  15. 2023Initializing Models with Larger Onescited by 39
  16. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  17. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 36
  18. 2021Curriculum-Meta Learning for Order-Robust Continual Relation Extractioncited by 34
  19. 2021Continual Learning for Recurrent Neural Networks: an Empirical Evaluationcited by 34
  20. 2022On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Modelscited by 34
  21. 2023Visual Tuningcited by 34
  22. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 33
  23. 2022Reversible Column Networkscited by 32
  24. 2022Learning Semantic Correspondence with Sparse Annotationscited by 31
  25. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
  26. 2024Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initializationcited by 29
  27. 2022Consistent Representation Learning for Continual Relation Extractioncited by 28
  28. 2021The Elastic Lottery Ticket Hypothesiscited by 23
  29. 2023BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformercited by 23
  30. 2023LEMON: Lossless model expansioncited by 23
  31. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  32. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Taskscited by 22
  33. 2024A survey of dynamic graph neural networkscited by 22
  34. 2021Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysiscited by 21
  35. 2021PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splittingcited by 20
  36. 2021Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimizationcited by 19
  37. 2022When, where, and how to add new neurons to ANNscited by 18
  38. 2022Efficient Architecture Search for Diverse Taskscited by 18

50 most cited of 246 All 246 (JSON)