mlsanityData to 7 Oct 2026

What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models

arXiv 2404.04759 ↗First posted 6 Apr 2024cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Compression techniques have been crucial in advancing machine learning by enabling efficient training and deployment of large-scale language models. However, these techniques have received limited attention in the context of low-resource language models, which are trained on even smaller amounts of data and under computational constraints, a scenario known as the "low-resource double-bind." This paper investigates the effectiveness of pruning, knowledge distillation, and quantization on an exclusively low-resourced, small-data language model, AfriBERTa. Through a battery of experiments, we assess the effects of compression on performance across several metrics beyond accuracy. Our study provides evidence that compression techniques significantly improve the efficiency and effectiveness of small-data language models, confirming that the prevailing beliefs regarding the effects of compression on large, heavily parameterized models hold true for less-parameterized, small-data models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 6 Apr 2024.

Builds on

22 of 30 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  12. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
10 more
  1. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  2. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  3. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  4. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  5. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  6. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  7. 2019Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLPcited by 73
  8. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  9. 2022MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognitioncited by 38
  10. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37

Built on by

Not cited yet.