mlsanityData to 7 Oct 2026

Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning

arXiv 2405.19462 ↗First posted 29 May 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Neural Machine Translation models are extremely data and compute-hungry. However, not all data points contribute equally to model training and generalization. Data pruning to remove the low-value data points has the benefit of drastically reducing the compute budget without significant drop in model performance. In this paper, we propose a new data pruning technique: Checkpoints Across Time (CAT), that leverages early model training dynamics to identify the most relevant data points for model performance. We benchmark CAT against several data pruning techniques including COMET-QE, LASER and LaBSE. We find that CAT outperforms the benchmarks on Indo-European languages on multiple test sets. When applied to English-German, English-French and English-Swahili translation tasks, CAT achieves comparable performance to using the full dataset, while pruning up to 50% of training data. We inspect the data points that CAT selects and find that it tends to favour longer sentences and sentences with unique or rare words.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 50 of its 83 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

50 of 83 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  10. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  11. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  12. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
38 more
  1. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  2. 2019The State of Sparsity in Deep Neural Networkscited by 471
  3. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  4. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  6. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  7. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  8. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  9. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  10. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  11. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
  12. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 133
  13. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 97
  14. 2020Estimating Example Difficulty Using Variance of Gradientscited by 96
  15. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  16. 2018On the Impact of Various Types of Noise on Neural Machine Translationcited by 76
  17. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  18. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  19. 2019Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translationcited by 56
  20. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  21. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  22. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  23. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  24. 2021Accelerating Deep Learning with Dynamic Data Pruningcited by 43
  25. 2018Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corporacited by 40
  26. 2018Effective Parallel Corpus Mining using Bilingual Sentence Embeddingscited by 39
  27. 2017Dynamic Data Selection for Neural Machine Translationcited by 33
  28. 2018Filtering and Mining Parallel Data in a Joint Multilingual Spacecited by 32
  29. 2020A Study of Gradient Variance in Deep Learningcited by 27
  30. 2021AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languagescited by 24
  31. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  32. 2020Parallel Corpus Filtering via Pre-trained Language Modelscited by 15
  33. 2023Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networkscited by 14
  34. 2022COMET-QE and Active Learning for Low-Resource Machine Translationcited by 10
  35. 2022Better Quality Estimation for Low Resource Corpus Miningcited by 8
  36. 2022Separating Grains from the Chaff: Using Data Filtering to Improve Multilingual Translation for Low-Resourced African Languagescited by 6
  37. 2022Cost-Effective Training in Low-Resource Neural Machine Translationcited by 2
  38. 2023"A Little is Enough": Few-Shot Quality Estimation based Corpus Filtering improves Machine Translationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 4.

10 papers.

  1. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  2. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  3. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  4. 2025The Multilingual Divide and Its Impact on Global AI Safetycited by 10
  5. 2024SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipecited by 3
  6. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 2
  7. 2025Emergence of Computational Structure in a Neural Network Physics Simulatorcited by 1
  8. 2026Robustness Emerges Early in Training Dynamics, but Is Not Preservedcited by 0
  9. 2026Synthetic Data Characterization via Training Dynamicscited by 0
  10. 2026Invent a Dataset: Measuring dataset generation abilities with zero seedcited by 0