# KIND: Knowledge Integration and Diversion for Training Decomposable Models

arXiv: https://arxiv.org/abs/2408.07337 · HTML: https://mlsanity.com/paper/2408.07337/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 52 references that are in the map.
- First posted on arXiv 14 Aug 2024.

## Builds on (most cited first, 30 of 37)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2304.01933/) (2023)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Generating Images with Sparse Representations](https://mlsanity.com/paper/2103.03841/) (2021)
- [DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://mlsanity.com/paper/2210.07558/) (2022)
- [Characterizing and Avoiding Negative Transfer](https://mlsanity.com/paper/1811.09751/) (2018)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [Compact Language Models via Pruning and Knowledge Distillation](https://mlsanity.com/paper/2407.14679/) (2024)
- [MiniViT: Compressing Vision Transformers with Weight Multiplexing](https://mlsanity.com/paper/2204.07154/) (2022)
- [IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis](https://mlsanity.com/paper/1807.06358/) (2018)
- [LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insights](https://mlsanity.com/paper/2404.11936/) (2024)
- [Few-Shot Adaptation of Generative Adversarial Networks](https://mlsanity.com/paper/2010.11943/) (2020)
- [LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models](https://mlsanity.com/paper/2404.11098/) (2024)
- [Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning](https://mlsanity.com/paper/2206.06122/) (2022)
- [IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2308.12043/) (2023)
- [Learngene: From Open-World to Your Learning Task](https://mlsanity.com/paper/2106.06788/) (2021)

## Built on by (most cited first, 10 of 10)

- [DivControl: Knowledge Diversion for Controllable Image Generation](https://mlsanity.com/paper/2507.23620/) (2025)
- [FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models](https://mlsanity.com/paper/2409.19289/) (2024)
- [SVasP: Self-Versatility Adversarial Style Perturbation for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2412.09073/) (2024)
- [A Unified Framework for Knowledge Transfer in Bidirectional Model Scaling](https://mlsanity.com/paper/2603.07506/) (2026)
- [FAD: Frequency Adaptation and Diversion for Cross-domain Few-shot Learning](https://mlsanity.com/paper/2505.08349/) (2025)
- [Knowledge Diversion for Efficient Morphology Control and Policy Transfer](https://mlsanity.com/paper/2512.09796/) (2025)
- [Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learning](https://mlsanity.com/paper/2504.19103/) (2025)
- [Self-Supervised Weight Templates for Scalable Vision Model Initialization](https://mlsanity.com/paper/2601.19694/) (2026)
- [Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform](https://mlsanity.com/paper/2603.07523/) (2026)
- [Bridging Body and Brain: Gene-Driven Morphology--Control Co-Design](https://mlsanity.com/paper/2609.31329/) (2026)

## Abstract (arXiv)

Pre-trained models have become the preferred backbone due to the increasing complexity of model parameters. However, traditional pre-trained models often face deployment challenges due to their fixed sizes, and are prone to negative transfer when discrepancies arise between training tasks and target tasks. To address this, we propose KIND, a novel pre-training method designed to construct decomposable models. KIND integrates knowledge by incorporating Singular Value Decomposition (SVD) as a structural constraint, with each basic component represented as a combination of a column vector, singular value, and row vector from U, Sigma, and V^top matrices. These components are categorized into learngenes for encapsulating class-agnostic knowledge and tailors for capturing class-specific knowledge, with knowledge diversion facilitated by a class gate mechanism during training. Extensive experiments demonstrate that models pre-trained with KIND can be decomposed into learngenes and tailors, which can be adaptively recombined for diverse resource-constrained deployments. Moreover, for tasks with large domain shifts, transferring only learngenes with task-agnostic knowledge, when combined with randomly initialized tailors, effectively mitigates domain shifts. Code will be made available at https://github.com/Te4P0t/KIND.