mlsanityData to 7 Oct 2026

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens

arXiv 2401.17377 ↗First posted 30 Jan 2024cs.CL cs.AI cs.IRcited by 148

Not on the map. Why?

Abstract (arXiv)

Are n-gram language models still relevant in this era of neural large language models (LLMs)? Our answer is yes, and we showcase their values in both text analysis and improving neural LLMs. This was done by modernizing n-gram LMs in two aspects. First, we train them at the same data scale as neural LLMs -- 5 trillion tokens. This is the largest n-gram LM ever built. Second, existing n-gram LMs use small n which hinders their performance; we instead allow n to be arbitrarily large, by introducing a new ∞-gram LM with backoff. Instead of pre-computing n-gram count tables (which would be very expensive), we develop an engine named infini-gram -- powered by suffix arrays -- that can compute ∞-gram (as well as n-gram with arbitrary n) probabilities with millisecond-level latency. The ∞-gram framework and infini-gram engine enable us to conduct many novel and interesting analyses of human-written and machine-generated text: we find that the ∞-gram LM has fairly high accuracy for next-token prediction (47%), and can complement neural LLMs to greatly reduce their perplexity. When analyzing machine-generated text, we also observe irregularities in the machine--∞-gram agreement level with respect to the suffix length, which indicates deficiencies in neural LLM pretraining and the positional embeddings of Transformers.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Builds on 25 of its 51 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

25 of 51 references.

References · 25

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  8. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
13 more
  1. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  2. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  3. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  4. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  5. 2023What's In My Big Data?cited by 129
  6. 2022Training Language Models with Memory Augmentationcited by 112
  7. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  8. 2022Nonparametric Masked Language Modelingcited by 39
  9. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 30
  10. 2023Data Portraits: Recording Foundation Model Training Datacited by 29
  11. 2023Copy Is All You Needcited by 25
  12. 2023Koala: An Index for Quantifying Overlaps with Pre-training Corporacited by 19
  13. 2019KiloGrams: Very Large N-Grams for Malware Classificationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 62; 2026: 53.

148 papers.

Sort:

By area:

  1. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  2. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  3. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  4. 2025SuperBPE: Space Travel for Language Modelscited by 58
  5. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  6. 2024Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrievalcited by 49
  7. 2024Can LLM Graph Reasoning Generalize beyond Pattern Memorization?cited by 46
  8. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 46
  9. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  10. 2024AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Textcited by 40
  11. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  12. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 33
38 more
  1. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  2. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  3. 2025OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokenscited by 27
  4. 2025WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMscited by 25
  5. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 24
  6. 2024Understanding Transformers via N-gram Statisticscited by 23
  7. 2024Avoiding Copyright Infringement via Large Language Model Unlearningcited by 22
  8. 2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 18
  9. 2024Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarkscited by 17
  10. 2024Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Biascited by 15
  11. 2025Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMscited by 15
  12. 2025The Illusion of Certainty: Uncertainty Quantification for LLMs Fails under Ambiguitycited by 15
  13. 2024From Loops to Oops: Fallback Behaviors of Language Models Under Uncertaintycited by 13
  14. 2024Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?cited by 13
  15. 2025Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Indexcited by 13
  16. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  17. 2026Speculative Speculative Decodingcited by 13
  18. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  19. 2024Can Transformers Learn n-gram Language Models?cited by 11
  20. 2024Probing Language Models on Their Knowledge Sourcecited by 11
  21. 2025Blackbox Model Provenance via Palimpsestic Membership Inferencecited by 11
  22. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  23. 2025Tracing Multilingual Factual Knowledge Acquisition in Pretrainingcited by 10
  24. 2024Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendationscited by 9
  25. 2025LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblancecited by 9
  26. 2025AI Agents as Universal Task Solverscited by 9
  27. 2024Attributing Culture-Conditioned Generations to Pretraining Corporacited by 8
  28. 2025Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Modelscited by 8
  29. 2025Truth over Tricks: Measuring and Mitigating Shortcut Learning in Misinformation Detectioncited by 8
  30. 2025Gaperon: A Peppered English-French Generative Language Model Suitecited by 8
  31. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Trainingcited by 8
  32. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 8
  33. 2024TAXI: Evaluating Categorical Knowledge Editing for Language Modelscited by 7
  34. 2025On the Acquisition of Shared Grammatical Representations in Bilingual Language Modelscited by 7
  35. 2025All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Languagecited by 7
  36. 2024The Role of n-gram Smoothing in the Age of Neural Networkscited by 6
  37. 2024Unlocking Legal Knowledge: A Multilingual Dataset for Judicial Summarization in Switzerlandcited by 6
  38. 2024N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needscited by 6

50 most cited of 148 All 148 (JSON)