oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes
Not on the map. Why?
Abstract (arXiv)
In this paper, we introduce the range of oBERTa language models, an easy-to-use set of language models which allows Natural Language Processing (NLP) practitioners to obtain between 3.8 and 24.3 times faster models without expertise in model compression. Specifically, oBERTa extends existing work on pruning, knowledge distillation, and quantization and leverages frozen embeddings improves distillation and model initialization to deliver higher accuracy on a broad range of transfer tasks. In generating oBERTa, we explore how the highly optimized RoBERTa differs from the BERT for pruning during pre-training and finetuning. We find it less amenable to compression during fine-tuning. We explore the use of oBERTa on seven representative NLP tasks and find that the improved compression techniques allow a pruned oBERTa model to match the performance of BERTbase and exceed the performance of Prune OFA Large on the SQUAD V1.1 Question Answering dataset, despite being 8x and 2x, respectively faster in inference. We release our code, training regimes, and associated model for broad usage to encourage usage and experimentation
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 30 Mar 2023.
Builds on
27 of 40 references.
References · 27
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
15 more
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
- 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
- 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
- 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
- 2019Q8BERT: Quantized 8Bit BERTcited by 296
- 2021Learned Token Pruning for Transformerscited by 138
- 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
- 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
- 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
- 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
- 2021How Well Do Sparse Imagenet Models Transfer?cited by 50
- 2020Least squares binary quantization of neural networkscited by 21
- 2022GMP*: Well-Tuned Gradual Magnitude Pruning Can Outperform Most BERT-Pruning Methodscited by 14
Built on by
Citing papers per year: 2025: 1.
1 paper.