mlsanityData to 14 Oct 2026

Is Modularity Transferable? A Case Study through the Lens of Knowledge Distillation

arXiv 2403.18804 ↗First posted 27 Mar 2024cs.CLcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

The rise of Modular Deep Learning showcases its potential in various Natural Language Processing applications. Parameter-efficient fine-tuning (PEFT) modularity has been shown to work for various use cases, from domain adaptation to multilingual setups. However, all this work covers the case where the modular components are trained and deployed within one single Pre-trained Language Model (PLM). This model-specific setup is a substantial limitation on the very modularity that modular architectures are trying to achieve. We ask whether current modular approaches are transferable between models and whether we can transfer the modules from more robust and larger PLMs to smaller ones. In this work, we aim to fill this gap via a lens of Knowledge Distillation, commonly used for model compression, and present an extremely straightforward approach to transferring pre-trained, task-specific PEFT modules between same-family PLMs. Moreover, we propose a method that allows the transfer of modules between incompatible PLMs without any change in the inference complexity. The experiments on Named Entity Recognition, Natural Language Inference, and Paraphrase Identification tasks over multiple languages and PEFT methods showcase the initial potential of transferable modularity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 13 of its 21 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

13 of 21 references.

References · 13

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  9. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  10. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  11. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 242
  12. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
1 more
  1. 2019How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representationscited by 63

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowledge Fusion of Large Language Models Via Modular SkillPacks
  2. 2023ModuleFormer: Modularity Emerges from Mixture-of-Experts
  3. 2023Unlocking Emergent Modularity in Large Language Models
  4. 2024m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers
  5. 2024Scalable Multi-Domain Adaptation of Language Models using Modular Experts
  6. 2023Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference
  7. 2024Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models
  8. 2024Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models