mlsanityData to 14 Oct 2026

Data Portraits: Recording Foundation Model Training Data

arXiv 2303.03919 ↗First posted 6 Mar 2023cs.LG cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

Foundation models are trained on increasingly immense and opaque datasets. Even while these models are now key in AI system building, it can be difficult to answer the straightforward question: has the model already encountered a given example during training? We therefore propose a widespread adoption of Data Portraits: artifacts that record training data and allow for downstream inspection. First we outline the properties of such an artifact and discuss how existing solutions can be used to increase transparency. We then propose and implement a solution based on data sketching, stressing fast and space efficient querying. Using our tools, we document a popular language modeling corpus (The Pile) and a recently released code modeling dataset (The Stack). We show that our solution enables answering questions about test set leakage and model plagiarism. Our tool is lightweight and fast, costing only 3% of the dataset size in overhead. We release a live interface of our tools at https://dataportraits.org/ and call on dataset and model creators to release Data Portraits as a complement to current documentation practices.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 20 of its 42 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

20 of 42 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  9. 2018Datasheets for Datasetscited by 1,386
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  12. 2018Model Cards for Model Reportingcited by 1,080
8 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  3. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  4. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  5. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  6. 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
  7. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 80
  8. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 14; 2025: 4; 2026: 3.

30 papers.

Sort:

By area:

  1. 2023StarCoder: may the source be with you!cited by 799
  2. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  3. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  4. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142
  5. 2023What's In My Big Data?cited by 129
  6. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  7. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 84
  8. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 56
  9. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 51
  10. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 43
  11. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  12. 2024Proving membership in LLM pretraining data via data watermarkscited by 40
18 more
  1. 2024AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledgecited by 37
  2. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  3. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  4. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  5. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  6. 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Datacited by 13
  7. 2024A Taxonomy for Data Contamination in Large Language Modelscited by 11
  8. 2024The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorizationcited by 6
  9. 2023Oasis: Data Curation and Assessment System for Pretraining of Large Language Modelscited by 3
  10. 2025TechOps: Technical Documentation Templates for the AI Actcited by 3
  11. 2026How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipelinecited by 3
  12. 2024CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkitcited by 2
  13. 2025Computational Budget Should Be Considered in Data Selectioncited by 2
  14. 2024What's New in My Data? Novelty Exploration via Contrastive Generationcited by 1
  15. 2025Certified Mitigation of Worst-Case LLM Copyright Infringementcited by 1
  16. 2026Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMscited by 1
  17. 2025On Finding Inconsistencies in Documentscited by 0
  18. 2026Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attackscited by 0