mlsanityData to 14 Oct 2026

SDBERT: SparseDistilBERT, a faster and smaller BERT model

arXiv 2208.10246 ↗First posted 28 Jul 2022cs.CL cs.IT cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this work we introduce a new transformer architecture called SparseDistilBERT (SDBERT), which is a combination of sparse attention and knowledge distillantion (KD). We implemented sparse attention mechanism to reduce quadratic dependency on input length to linear. In addition to reducing computational complexity of the model, we used knowledge distillation (KD). We were able to reduce the size of BERT model by 60% while retaining 97% performance and it only took 40% of time to train.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 6 references that are in the map. First posted on arXiv 28 Jul 2022.

Builds on

5 of 6 references.

References · 5

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  5. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
  2. 2022Sparse*BERT: Sparse Models Generalize To New tasks and Domains
  3. 2021RefBERT: Compressing BERT by Referencing to Pre-computed Representations
  4. 2021Prune Once for All: Sparse Pre-Trained Language Models
  5. 2021KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
  6. 2021Learning to Augment for Data-Scarce Domain BERT Knowledge Distillation
  7. 2021Marginal Utility Diminishes: Exploring the Minimum Knowledge for BERT Knowledge Distillation
  8. 2022BiBERT: Accurate Fully Binarized BERT