mlsanityData to 14 Oct 2026

One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

arXiv 1312.3005 ↗First posted 11 Dec 2013cs.CLcited by 308

Not on the map. Why?

Cited by 84 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a new benchmark corpus to be used for measuring progress in statistical language modeling. With almost one billion words of training data, we hope this benchmark will be useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques. We show performance of several well-known types of language models, with the best results achieved with a recurrent neural network based language model. The baseline unpruned Kneser-Ney 5-gram model achieves perplexity 67.6; a combination of techniques leads to 35% reduction in perplexity, or 10% reduction in cross-entropy (bits), over that baseline. The benchmark is available as a code.google.com project; besides the scripts needed to rebuild the training/held-out data, it also makes available log-probability values for each word in each of ten held-out data sets, for each of the baseline n-gram models.

Key facts. Cited by 308 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Dec 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2017: 1; 2018: 1; 2019: 0; 2020: 1; 2021: 49; 2022: 46; 2023: 42; 2024: 44; 2025: 48; 2026: 76.

308 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  6. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  9. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  10. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  11. 2023The Falcon Series of Open Language Modelscited by 535
  12. 2023Symbolic Discovery of Optimization Algorithmscited by 434
38 more
  1. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  2. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  3. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  4. 2022Long Range Language Modeling via Gated State Spacescited by 265
  5. 2021Hash Layers For Large Sparse Modelscited by 237
  6. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  7. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  8. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  9. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  10. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  11. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  12. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  13. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  14. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 155
  15. 2023Likelihood-Based Diffusion Language Modelscited by 151
  16. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  17. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  18. 2025The Diffusion Dualitycited by 135
  19. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  20. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  21. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  22. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  23. 2024How to Train Data-Efficient LLMscited by 93
  24. 2022Evaluating Human-Language Model Interactioncited by 92
  25. 2021SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarioscited by 84
  26. 2024Fisher Flow Matching for Generative Modeling over Discrete Datacited by 72
  27. 2023MultiLegalPile: A 689GB Multilingual Legal Corpuscited by 60
  28. 2021Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Modelscited by 57
  29. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  30. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 57
  31. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 54
  32. 2021H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequencescited by 47
  33. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  34. 2021Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairscited by 45
  35. 2026Categorical Flow Mapscited by 44
  36. 2025Implicit Reasoning in Large Language Models: A Comprehensive Surveycited by 43
  37. 2021Mitigating harm in language models with conditional-likelihood filtrationcited by 40
  38. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 40

50 most cited of 308 All 308 (JSON)