mlsanityData to 14 Oct 2026

BERT on a Data Diet: Finding Important Examples by Gradient-Based Pruning

arXiv 2211.05610 ↗First posted 10 Nov 2022cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

Current pre-trained language models rely on large datasets for achieving state-of-the-art performance. However, past research has shown that not all examples in a dataset are equally important during training. In fact, it is sometimes possible to prune a considerable fraction of the training set while maintaining the test performance. Established on standard vision benchmarks, two gradient-based scoring metrics for finding important examples are GraNd and its estimated version, EL2N. In this work, we employ these two metrics for the first time in NLP. We demonstrate that these metrics need to be computed after at least one epoch of fine-tuning and they are not reliable in early steps. Furthermore, we show that by pruning a small portion of the examples with the highest GraNd/EL2N scores, we can not only preserve the test accuracy, but also surpass it. This paper details adjustments and implementation choices which enable GraNd and EL2N to be applied to NLP.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

13 of 17 references.

References · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  8. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  9. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  10. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  11. 2019An Effective Label Noise Model for DNN Text Classificationcited by 22
  12. 2021Clean or Annotate: How to Spend a Limited Data Collection Budgetcited by 10
1 more
  1. 2022Let the Model Decide its Curriculum for Multitask Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 6; 2025: 4; 2026: 1.

17 papers.

Sort:

By area:

  1. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 69
  2. 2023PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuningcited by 29
  3. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 20
  4. 2023Data Management For Training Large Language Models: A Surveycited by 17
  5. 2024SMART: Submodular Data Mixture Strategy for Instruction Tuningcited by 17
  6. 2023Influence Scores at Scale for Efficient Language Data Samplingcited by 5
  7. 2025Swift Cross-Dataset Pruning: Enhancing Fine-Tuning Efficiency in Natural Language Understandingcited by 4
  8. 2025A Survey of LLM × DATAcited by 4
  9. 2024Efficient Backpropagation with Variance-Controlled Adaptive Samplingcited by 3
  10. 2024A Bayesian Approach to Data Point Selectioncited by 3
  11. 2023Efficient Architecture Search via Bi-level Data Pruningcited by 2
  12. 2024Sexism Detection on a Data Dietcited by 2
5 more
  1. 2024MelissaDL x Breed: Towards Data-Efficient On-line Supervised Training of Multi-parametric Surrogates with Active Learningcited by 1
  2. 2024Automatic Pruning of Fine-tuning Datasets for Transformer-based Language Modelscited by 0
  3. 2025As easy as PIE: understanding when pruning causes language models to disagreecited by 0
  4. 2025Efficient Training of Deep Networks using Guided Spectral Data Selection: A Step Toward Learning What You Needcited by 0
  5. 2026Position: Stop Preaching and Start Practising Data Frugality for Responsible Development of AIcited by 0