mlsanityData to 7 Oct 2026

Koala: An Index for Quantifying Overlaps with Pre-training Corpora

arXiv 2303.14770 ↗First posted 26 Mar 2023cs.CL cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

In very recent years more attention has been placed on probing the role of pre-training data in Large Language Models (LLMs) downstream behaviour. Despite the importance, there is no public tool that supports such analysis of pre-training corpora at large scale. To help research in this space, we launch Koala, a searchable index over large pre-training corpora using compressed suffix arrays with highly efficient compression rate and search support. In its first release we index the public proportion of OPT 175B pre-training data. Koala provides a framework to do forensic analysis on the current and future benchmarks as well as to assess the degree of memorization in the output from the LLMs. Koala is available for public use at https://koala-index.erc.monash.edu/.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 26 Mar 2023.

Builds on

31 of 47 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
19 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  7. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2020Recipes for building an open-domain chatbotcited by 593
  10. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  11. 2022Data Contamination: From Memorization to Exploitationcited by 234
  12. 2018A Simple Method for Commonsense Reasoningcited by 223
  13. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  14. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  15. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  16. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 137
  17. 2021Memorisation versus Generalisation in Pre-trained Language Modelscited by 32
  18. 2013From Theory to Practice: Plug and Play with Succinct Data Structurescited by 6
  19. 2016Fast, Small and Exact: Infinite-order Language Modelling with Compressed Suffix Treescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 8; 2026: 1.

19 papers.

Sort:

By area:

  1. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  2. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  3. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  4. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  5. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 48
  6. 2024ConStat: Performance-Based Contamination Detection in Large Language Modelscited by 33
  7. 2024Evading Data Contamination Detection for Language Models is (too) Easycited by 30
  8. 2025D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuningcited by 21
  9. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  10. 2024Language model developers should report train-test overlapcited by 16
  11. 2024Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluatorscited by 13
  12. 2025Importance-Aware Data Selection for Efficient LLM Instruction Tuningcited by 7
7 more
  1. 2025Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Processcited by 4
  2. 2025RICo: Refined In-Context Contribution for Automatic Instruction-Tuning Data Selectioncited by 2
  3. 2025ProDS: Preference-oriented Data Selection for Instruction Tuningcited by 1
  4. 2025Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Studycited by 1
  5. 2025Data Selection for LLM Alignment Using Fine-Grained Preferencescited by 1
  6. 2025Do Generalisation Results Generalise?cited by 0
  7. 2026Provable Joint Decontamination for Benchmarking Multiple Large Language Modelscited by 0