mlsanityData to 7 Oct 2026

CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillation

arXiv 2204.07674 ↗First posted 15 Apr 2022cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation (KD) is an efficient framework for compressing large-scale pre-trained language models. Recent years have seen a surge of research aiming to improve KD by leveraging Contrastive Learning, Intermediate Layer Distillation, Data Augmentation, and Adversarial Training. In this work, we propose a learning based data augmentation technique tailored for knowledge distillation, called CILDA. To the best of our knowledge, this is the first time that intermediate layer representations of the main task are used in improving the quality of augmented samples. More precisely, we introduce an augmentation technique for KD based on intermediate layer matching using contrastive loss to improve masked adversarial data augmentation. CILDA outperforms existing state-of-the-art KD approaches on the GLUE benchmark, as well as in an out-of-domain evaluation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 15 Apr 2022.

Builds on

39 of 47 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
27 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  6. 2019Contrastive Representation Distillationcited by 734
  7. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  9. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  10. 2019Revealing the Dark Secrets of BERTcited by 300
  11. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  12. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  13. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  14. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  15. 2020ALP-KD: Attention-Based Layer Projection for Knowledge Distillationcited by 65
  16. 2021Annealing Knowledge Distillationcited by 61
  17. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  18. 2020CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understandingcited by 41
  19. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  20. 2020LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understandingcited by 27
  21. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  22. 2020Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layerscited by 24
  23. 2020Towards Zero-Shot Knowledge Distillation for Natural Language Processingcited by 20
  24. 2021Pro-KD: Progressive Distillation by Following the Footsteps of the Teachercited by 16
  25. 2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
  26. 2022When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentationcited by 5
  27. 2021How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understandingcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 0; 2025: 1.

4 papers.

  1. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
  2. 2023Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentationcited by 7
  3. 2025Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selectioncited by 4
  4. 2023HARD: Hard Augmentations for Robust Distillationcited by 1