# BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distance

arXiv: https://arxiv.org/abs/2010.06133 · HTML: https://mlsanity.com/paper/2010.06133/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2020.

## Built on by (most cited first, 27 of 27)

- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2307.15190/) (2023)
- [Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification](https://mlsanity.com/paper/2109.05427/) (2021)
- [Dynamic Knowledge Distillation for Pre-trained Language Models](https://mlsanity.com/paper/2109.11295/) (2021)
- [MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation](https://mlsanity.com/paper/2204.07675/) (2022)
- [Lifting the Curse of Capacity Gap in Distilling Language Models](https://mlsanity.com/paper/2305.12129/) (2023)
- [A Comprehensive Survey of Compression Algorithms for Language Models](https://mlsanity.com/paper/2401.15347/) (2024)
- [ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques](https://mlsanity.com/paper/2103.11367/) (2021)
- [From Discourse to Narrative: Knowledge Projection for Event Relation Extraction](https://mlsanity.com/paper/2106.08629/) (2021)
- [Sparse Teachers Can Be Dense with Knowledge](https://mlsanity.com/paper/2210.03923/) (2022)
- [ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression](https://mlsanity.com/paper/2106.02241/) (2021)
- [Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing](https://mlsanity.com/paper/2109.11105/) (2021)
- [Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment](https://mlsanity.com/paper/2411.01547/) (2024)
- [Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation](https://mlsanity.com/paper/2106.05691/) (2021)
- [Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Models](https://mlsanity.com/paper/2303.09639/) (2023)
- [A Comparative Analysis of Task-Agnostic Distillation Methods for Compressing Transformer Language Models](https://mlsanity.com/paper/2310.08797/) (2023)
- [MiniDisc: Minimal Distillation Schedule for Language Model Compression](https://mlsanity.com/paper/2205.14570/) (2022)
- [Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective](https://mlsanity.com/paper/2302.01530/) (2023)
- [Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models](https://mlsanity.com/paper/2306.08909/) (2023)
- [rm SP3: Enhancing Structured Pruning via PCA Projection](https://mlsanity.com/paper/2308.16475/) (2023)
- [Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT](https://mlsanity.com/paper/2203.09055/) (2022)
- [Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search](https://mlsanity.com/paper/2107.07173/) (2021)
- [Reminding the Incremental Language Model via Data-Free Self-Distillation](https://mlsanity.com/paper/2110.08745/) (2021)
- [Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention](https://mlsanity.com/paper/2310.07911/) (2023)
- [MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models](https://mlsanity.com/paper/2407.02775/) (2024)
- [HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models](https://mlsanity.com/paper/2504.17449/) (2025)
- [Basic Reading Distillation](https://mlsanity.com/paper/2507.19741/) (2025)

## Abstract (arXiv)

Pre-trained language models (e.g., BERT) have achieved significant success in various natural language processing (NLP) tasks. However, high storage and computational costs obstruct pre-trained language models to be effectively deployed on resource-constrained devices. In this paper, we propose a novel BERT distillation method based on many-to-many layer mapping, which allows each intermediate student layer to learn from any intermediate teacher layers. In this way, our model can learn from different teacher layers adaptively for various NLP tasks. %motivated by the intuition that different NLP tasks require different levels of linguistic knowledge contained in the intermediate layers of BERT. In addition, we leverage Earth Mover's Distance (EMD) to compute the minimum cumulative cost that must be paid to transform knowledge from teacher network to student network. EMD enables the effective matching for many-to-many layer mapping. %EMD can be applied to network layers with different sizes and effectively measures semantic distance between the teacher network and student network. Furthermore, we propose a cost attention mechanism to learn the layer weights used in EMD automatically, which is supposed to further improve the model's performance and accelerate convergence time. Extensive experiments on GLUE benchmark demonstrate that our model achieves competitive performance compared to strong competitors in terms of both accuracy and model compression.