mlsanityData to 7 Oct 2026

Causal Distillation for Language Models

arXiv 2112.02505 ↗First posted 5 Dec 2021cs.CL cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Distillation efforts have led to language models that are more compact and efficient without serious drops in performance. The standard approach to distillation trains a student model against two objectives: a task-specific objective (e.g., language modeling) and an imitation objective that encourages the hidden states of the student model to be similar to those of the larger teacher model. In this paper, we show that it is beneficial to augment distillation with a third objective that encourages the student to imitate the causal computation process of the teacher through interchange intervention training(IIT). IIT pushes the student model to become a causal abstraction of the teacher model - a simpler model with the same causal structure. IIT is fully differentiable, easily implemented, and combines flexibly with other objectives. Compared with standard distillation of BERT, distillation via IIT results in lower perplexity on Wikipedia (masked language modeling) and marked improvements on the GLUE benchmark (natural language understanding), SQuAD (question answering), and CoNLL-2003 (named entity recognition).

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 5 Dec 2021.

Builds on

22 of 34 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  8. 2018Universal Transformerscited by 613
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  10. 2021Causal Abstractions of Neural Networkscited by 401
  11. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  12. 2013Direct and Indirect Effectscited by 276
10 more
  1. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  2. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  3. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  4. 2018Abstracting Causal Modelscited by 104
  5. 2020Analysing Lexical Semantic Change with Contextualised Word Representationscited by 92
  6. 2019A Tensorized Transformer for Language Modelingcited by 85
  7. 2019Approximate Causal Abstractioncited by 62
  8. 2020Causal Effects of Linguistic Propertiescited by 39
  9. 2019Structured Pruning of a BERT-based Question Answering Modelcited by 16
  10. 2021Model Compression for Domain Adaptation through Causal Effect Estimationcited by 6

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 8; 2024: 5; 2025: 3.

19 papers.

Sort:

By area:

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  4. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  5. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  6. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  7. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  8. 2023Dissociating language and thought in large language modelscited by 27
  9. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  10. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  11. 2024Do Large Language Models Mirror Cognitive Language Processing?cited by 9
  12. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
7 more
  1. 2024HLQ: Fast and Efficient Backpropagation via Hadamard Low-rank Quantizationcited by 2
  2. 2025Causal Distillation: Transferring Structured Explanations from Large to Compact Language Modelscited by 2
  3. 2023Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddingscited by 1
  4. 2023Semantic Composition in Visually Grounded Language Modelscited by 1
  5. 2023Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Methodcited by 0
  6. 2024Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Modelscited by 0
  7. 2025Inducing Causal Structure for Interpretable Neural Networks Applied to Glucose Prediction for T1DM Patientscited by 0