# Learngene: Inheriting Condensed Knowledge from the Ancestry Model to Descendant Models

arXiv: https://arxiv.org/abs/2305.02279 · HTML: https://mlsanity.com/paper/2305.02279/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 104 references that are in the map.
- First posted on arXiv 3 May 2023.

## Builds on (most cited first, 30 of 66)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)

## Built on by (most cited first, 24 of 24)

- [WAVE: Weight Templates for Adaptive Initialization of Variable-sized Models](https://mlsanity.com/paper/2406.17503/) (2024)
- [Transformer as Linear Expansion of Learngene](https://mlsanity.com/paper/2312.05614/) (2023)
- [Transferring Core Knowledge via Learngenes](https://mlsanity.com/paper/2401.08139/) (2024)
- [KIND: Knowledge Integration and Diversion for Training Decomposable Models](https://mlsanity.com/paper/2408.07337/) (2024)
- [Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Models](https://mlsanity.com/paper/2404.16897/) (2024)
- [Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models](https://mlsanity.com/paper/2312.10104/) (2023)
- [DivControl: Knowledge Diversion for Controllable Image Generation](https://mlsanity.com/paper/2507.23620/) (2025)
- [FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models](https://mlsanity.com/paper/2409.19289/) (2024)
- [One-Shot Knowledge Transfer for Scalable Person Re-Identification](https://mlsanity.com/paper/2511.06016/) (2025)
- [Stratified Knowledge-Density Super-Network for Scalable Vision Transformers](https://mlsanity.com/paper/2511.11683/) (2025)
- [VT-Bench: A Unified Benchmark for Visual-Tabular Multi-Modal Learning](https://mlsanity.com/paper/2605.08146/) (2026)
- [KALAHash: Knowledge-Anchored Low-Resource Adaptation for Deep Hashing](https://mlsanity.com/paper/2412.19417/) (2024)
- [Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions](https://mlsanity.com/paper/2605.07271/) (2026)
- [Knowledge Diversion for Efficient Morphology Control and Policy Transfer](https://mlsanity.com/paper/2512.09796/) (2025)
- [SafeGene: Reusable Adapters for Transferable Safety Alignment](https://mlsanity.com/paper/2606.06519/) (2026)
- [Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learning](https://mlsanity.com/paper/2504.19103/) (2025)
- [Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones](https://mlsanity.com/paper/2506.05641/) (2025)
- [Self-Supervised Weight Templates for Scalable Vision Model Initialization](https://mlsanity.com/paper/2601.19694/) (2026)
- [CLIP-Map: Structured Matrix Mapping for Parameter-Efficient CLIP Compression](https://mlsanity.com/paper/2602.05909/) (2026)
- [Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform](https://mlsanity.com/paper/2603.07523/) (2026)
- [Constraint-based Pre-training: From Structured Constraints to Scalable Model Initialization](https://mlsanity.com/paper/2604.14769/) (2026)
- [Chain-based Distillation for Effective Initialization of Variable-Sized Small Language Models](https://mlsanity.com/paper/2605.07783/) (2026)
- [Learngene Search Across Multiple Datasets for Building Variable-Sized Models](https://mlsanity.com/paper/2605.08209/) (2026)
- [XPERT: Expert Knowledge Transfer for Effective Training of Language Models](https://mlsanity.com/paper/2605.08842/) (2026)

## Abstract (arXiv)

During the continuous evolution of one organism's ancestry, its genes accumulate extensive experiences and knowledge, enabling newborn descendants to rapidly adapt to their specific environments. Motivated by this observation, we propose a novel machine learning paradigm Learngene to enable learning models to incorporate three key characteristics of genes. (i) Accumulating: the knowledge is accumulated during the continuous learning of an ancestry model. (ii) Condensing: the extensive accumulated knowledge is condensed into a much more compact information piece, i.e., learngene. (iii) Inheriting: the condensed learngene is inherited to make it easier for descendant models to adapt to new environments. Since accumulating has been studied in well-established paradigms like large-scale pre-training and lifelong learning, we focus on condensing and inheriting, which induces three key issues and we provide the preliminary solutions to these issues in this paper: (i) Learngene Form: the learngene is set to a few integral layers that can preserve significance. (ii) Learngene Condensing: we identify which layers among the ancestry model have the most similarity as one pseudo descendant model. (iii) Learngene Inheriting: to construct distinct descendant models for the specific downstream tasks, we stack some randomly initialized layers to the learngene layers. Extensive experiments across various settings, including using different network architectures like Vision Transformer (ViT) and Convolutional Neural Networks (CNNs) on different datasets, are carried out to confirm four advantages of Learngene: it makes the descendant models 1) converge more quickly, 2) exhibit less sensitivity to hyperparameters, 3) perform better, and 4) require fewer training samples to converge.