mlsanityData to 7 Oct 2026

Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics

arXiv 2009.10795 ↗First posted 22 Sep 2020cs.CLcited by 423

Not on the map. Why?

Abstract (arXiv)

Large datasets have become commonplace in NLP research. However, the increased emphasis on data quantity has made it challenging to assess the quality of data. We introduce Data Maps---a model-based tool to characterize and diagnose datasets. We leverage a largely ignored source of information: the behavior of the model on individual instances during training (training dynamics) for building data maps. This yields two intuitive measures for each example---the model's confidence in the true class, and the variability of this confidence across epochs---obtained in a single run of training. Experiments across four datasets show that these model-dependent measures reveal three distinct regions in the data map, each with pronounced characteristics. First, our data maps show the presence of "ambiguous" regions with respect to the model, which contribute the most towards out-of-distribution generalization. Second, the most populous regions in the data are "easy to learn" for the model, and play an important role in model optimization. Finally, data maps uncover a region with instances that the model finds "hard to learn"; these often correspond to labeling errors. Our results indicate that a shift in focus from quantity to quality of data could lead to robust models and improved out-of-distribution generalization.

Key facts. Cited by 423 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 70; 2023: 98; 2024: 81; 2025: 67; 2026: 69.

423 papers.

Sort:

By area:

  1. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  2. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  3. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  4. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  5. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  7. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  8. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  9. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  10. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  11. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  12. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 115
38 more
  1. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  2. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  3. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  4. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  5. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  6. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  7. 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
  8. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
  9. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  10. 2022MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflictscited by 81
  11. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  12. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  13. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  14. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  15. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
  16. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  17. 2023Contrasting Linguistic Patterns in Human and LLM-Generated News Textcited by 62
  18. 2022Handling Bias in Toxic Speech Detection: A Surveycited by 60
  19. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 53
  20. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 52
  21. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  22. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  23. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  24. 2022Annotation Error Detection: Analyzing the Past and Present for a More Coherent Futurecited by 45
  25. 2022Rethinking Attention-Model Explainability through Faithfulness Violation Testcited by 43
  26. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  27. 2024VariErr NLI: Separating Annotation Error from Human Label Variationcited by 42
  28. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  29. 2024The Unreasonable Effectiveness of Easy Training Data for Hard Taskscited by 40
  30. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  31. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  32. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  33. 2023Data Valuation Without Training of a Modelcited by 35
  34. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 34
  35. 2024Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Modelscited by 32
  36. 2022On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliencycited by 31
  37. 2023Ecosystem Graphs: The Social Footprint of Foundation Modelscited by 31
  38. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30

50 most cited of 423 All 423 (JSON)