mlsanityData to 14 Oct 2026

Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets

arXiv 2302.13959 ↗First posted 27 Feb 2023cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Increasingly larger datasets have become a standard ingredient to advancing the state-of-the-art in NLP. However, data quality might have already become the bottleneck to unlock further gains. Given the diversity and the sizes of modern datasets, standard data filtering is not straight-forward to apply, because of the multifacetedness of the harmful data and elusiveness of filtering rules that would generalize across multiple tasks. We study the fitness of task-agnostic self-influence scores of training examples for data cleaning, analyze their efficacy in capturing naturally occurring outliers, and investigate to what extent self-influence based data cleaning can improve downstream performance in machine translation, question answering and text classification, building up on recent approaches to self-influence calculation and automated curriculum learning.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 46 of its 66 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

46 of 66 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  8. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  9. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  10. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  11. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  12. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
34 more
  1. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  2. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  3. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
  4. 2017Automated Curriculum Learning for Neural Networkscited by 257
  5. 2020Influence Functions in Deep Learning Are Fragilecited by 252
  6. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  7. 2016Ex Machina: Personal Attacks Seen at Scalecited by 210
  8. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  9. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 202
  10. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  11. 2017Deep Learning is Robust to Massive Label Noisecited by 165
  12. 2019SELF: Learning to Filter Noisy Labels with Self-Ensemblingcited by 164
  13. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  14. 2018Stronger Data Poisoning Attacks Break Data Sanitization Defensescited by 135
  15. 2021Scaling Up Influence Functionscited by 132
  16. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  17. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  18. 2018On the Impact of Various Types of Noise on Neural Machine Translationcited by 76
  19. 2018What Makes Reading Comprehension Questions Easier?cited by 60
  20. 2018Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Taskcited by 56
  21. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  22. 2018Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corporacited by 40
  23. 2021Revisiting Methods for Finding Influential Examplescited by 39
  24. 2019Improving Robustness of Machine Translation with Synthetic Noisecited by 36
  25. 2022First is Better Than Last for Language Data Influencecited by 34
  26. 2017Dynamic Data Selection for Neural Machine Translationcited by 33
  27. 2019Reinforcement Learning based Curriculum Optimization for Neural Machine Translationcited by 29
  28. 2020RelatIF: Identifying Explanatory Training Examples via Relative Influencecited by 28
  29. 2019Question Answering is a Format; When is it Useful?cited by 25
  30. 2017How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?cited by 23
  31. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16
  32. 2019Mitigating Noisy Inputs for Question Answeringcited by 6
  33. 2022Analyzing the Use of Influence Functions for Instance-Specific Data Filtering in Neural Machine Translationcited by 3
  34. 2022Understanding Text Classification Data and Models Using Aggregated Input Saliencecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 10; 2026: 3.

19 papers.

Sort:

By area:

  1. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 113
  2. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
  3. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  4. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 26
  5. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  6. 2024Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Modelscited by 10
  7. 2024Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methodscited by 9
  8. 2025First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimationcited by 5
  9. 2025Disentangling the Roles of Representation and Selection in Data Pruningcited by 4
  10. 2025Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuningcited by 4
  11. 2026Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMscited by 4
  12. 2025Unraveling Indirect In-Context Learning Using Influence Functionscited by 3
7 more
  1. 2025Influence-driven Curriculum Learning for Pre-training on Limited Datacited by 2
  2. 2025Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Modelscited by 2
  3. 2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiencescited by 1
  4. 2024Dynamic Skill Adaptation for Large Language Modelscited by 1
  5. 2026Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models Via Layer Gradient Analysiscited by 1
  6. 2025What Is The Performance Ceiling of My Classifier? Utilizing Category-Wise Influence Functions for Pareto Frontier Analysiscited by 0
  7. 2026Support or Magnitude? Characterizing Neuron Relevance for Instruction Data Selectioncited by 0