mlsanityData to 7 Oct 2026

Contrastive Distillation on Intermediate Representations for Language Model Compression

arXiv 2009.14167 ↗First posted 29 Sep 2020cs.CL cs.LGcited by 57

Not on the map. Why?

Abstract (arXiv)

Existing language model compression methods mostly use a simple L2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one. Although widely used, this objective by design assumes that all the dimensions of hidden representations are independent, failing to capture important structural knowledge in the intermediate layers of the teacher network. To achieve better distillation efficacy, we propose Contrastive Distillation on Intermediate Representations (CoDIR), a principled knowledge distillation framework where the student is trained to distill knowledge through intermediate layers of the teacher via a contrastive objective. By learning to distinguish positive sample from a large set of negative samples, CoDIR facilitates the student's exploitation of rich information in teacher's hidden layers. CoDIR can be readily applied to compress large-scale language models in both pre-training and finetuning stages, and achieves superb performance on the GLUE benchmark, outperforming state-of-the-art compression methods.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 10; 2023: 13; 2024: 7; 2025: 9; 2026: 5.

57 papers.

Sort:

By area:

  1. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  2. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
  3. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  4. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  5. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  6. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  7. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  8. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 74
  9. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  10. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
  11. 2021Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classificationcited by 43
  12. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
38 more
  1. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  2. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  3. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
  4. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 28
  5. 2023AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compressioncited by 26
  6. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  7. 2023PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillationcited by 25
  8. 2022Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Taskscited by 22
  9. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillationcited by 19
  10. 2021Rethinking InfoNCE: How Many Negative Samples Do You Need?cited by 18
  11. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 18
  12. 2023One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Modelscited by 16
  13. 2024Search for Efficient Large Language Modelscited by 13
  14. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  15. 2021Finding the Right Moment: Human-Assisted Trailer Creation via Task Compositioncited by 12
  16. 2021Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processingcited by 11
  17. 2023Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLMcited by 11
  18. 2023How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectivescited by 9
  19. 2024ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantizationcited by 8
  20. 2022Knowledge Distillation of Russian Language Models with Reduction of Vocabularycited by 7
  21. 2022Distilling Multi-Scale Knowledge for Event Temporal Relation Extractioncited by 6
  22. 2024MoreauPruner: Robust Pruning of Large Language Models against Weight Perturbationscited by 5
  23. 2022CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillationcited by 4
  24. 2025Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matchingcited by 4
  25. 2023Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compressioncited by 3
  26. 2023Text Representation Distillation via Information Bottleneck Principlecited by 3
  27. 2026NITP: Next Implicit Token Prediction for LLM Pre-trainingcited by 3
  28. 2021DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddingscited by 2
  29. 2022Do we need Label Regularization to Fine-tune Pre-trained Language Models?cited by 2
  30. 20223D-Augmented Contrastive Knowledge Distillation for Image-based Object Pose Estimationcited by 2
  31. 2022Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddingscited by 2
  32. 2025EvoP: Robust LLM Inference via Evolutionary Pruningcited by 2
  33. 2025PIP: Perturbation-based Iterative Pruning for Large Language Modelscited by 1
  34. 2025Structure-Aware Automatic Channel Pruning by Searching with Graph Embeddingcited by 1
  35. 2025Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruningcited by 1
  36. 2026DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generationcited by 1
  37. 2023Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Methodcited by 0
  38. 2023Continual Contrastive Spoken Language Understandingcited by 0

50 most cited of 57 All 57 (JSON)