mlsanityData to 14 Oct 2026

Paloma: A Benchmark for Evaluating Language Model Fit

arXiv 2312.10523 ↗First posted 16 Dec 2023cs.CL cs.AI cs.LGcited by 54

Not on the map. Why?

Abstract (arXiv)

Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training. Implicitly or explicitly, this data is composed of domains--varying distributions of language. We introduce Perplexity Analysis for Language Model Assessment (Paloma), a benchmark to measure LM fit to 546 English and code domains, instead of assuming perplexity on one distribution extrapolates to others. We include two new datasets of the top 100 subreddits (e.g., r/depression on Reddit) and programming languages (e.g., Java on GitHub), both sources common in contemporary LMs. With our benchmark, we release 6 baseline 1B LMs carefully controlled to provide fair comparisons about which pretraining corpus is best and code for others to apply those controls to their own experiments. Our case studies demonstrate how the fine-grained results from Paloma surface findings such as that models pretrained without data beyond Common Crawl exhibit anomalous gaps in LM fit to many domains or that loss is dominated by the most frequently occurring strings in the vocabulary.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 52 of its 87 references that are in the map. First posted on arXiv 16 Dec 2023.

Builds on

52 of 87 references.

References · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
38 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  8. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  12. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  13. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  14. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  15. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  16. 2022Predictability and Surprise in Large Generative Modelscited by 210
  17. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  18. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 196
  19. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  20. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
  21. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  22. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  23. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  24. 2023What's In My Big Data?cited by 129
  25. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  26. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  27. 2022Training Trajectories of Language Models Across Scalescited by 81
  28. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  29. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  30. 2018A Web-scale system for scientific knowledge explorationcited by 39
  31. 2021Efficient Hierarchical Domain Adaptation for Pretrained Language Modelscited by 32
  32. 2021You should evaluate your language model on marginal likelihood over tokenisationscited by 30
  33. 2022M2D2: A Massively Multi-domain Language Modeling Datasetcited by 26
  34. 2018What is Gab? A Bastion of Free Speech or an Alt-Right Echo Chamber?cited by 24
  35. 2020Raiders of the Lost Kek: 3.5 Years of Augmented 4chan Posts from the Politically Incorrect Boardcited by 24
  36. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  37. 2023Should you marginalize over possible tokenizations?cited by 18
  38. 2023Scaling Laws Do Not Scalecited by 13

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 21; 2026: 14.

54 papers.

Sort:

By area:

  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  2. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  4. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  5. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  6. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  7. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  8. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  9. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 56
  10. 2024Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compressioncited by 33
  11. 2025Scaling Laws Are Unreliable for Downstream Tasks: A Reality Checkcited by 29
  12. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 26
38 more
  1. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 26
  2. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 25
  3. 2024Data Contamination Report from the 2024 CONDA Shared Taskcited by 20
  4. 2024Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarkscited by 17
  5. 2024From Single to Multi: How LLMs Hallucinate in Multi-Document Summarizationcited by 17
  6. 2024Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Biascited by 15
  7. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  8. 2024Tele-FLM Technical Reportcited by 12
  9. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  10. 2024Scalable Data Ablation Approximations for Language Models through Modular Training and Mergingcited by 11
  11. 2025Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completenesscited by 11
  12. 2025Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Codecited by 10
  13. 2025Unlocking Post-hoc Dataset Inference with Synthetic Datacited by 7
  14. 2025Exploiting Vocabulary Frequency Imbalance in Language Model Pre-trainingcited by 7
  15. 2024Measuring Fingerprints of Web-filtered Text Datasets and Fingerprint Propagation Through Trainingcited by 5
  16. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 5
  17. 2025BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Modelscited by 5
  18. 2025Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learningcited by 5
  19. 2025Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Modelcited by 4
  20. 2024From Intentions to Techniques: A Comprehensive Taxonomy and Challenges in Text Watermarking for Large Language Modelscited by 3
  21. 2025ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data Selectioncited by 3
  22. 2025Learning Dynamics of Meta-Learning in Small Model Pretrainingcited by 3
  23. 2025Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languagescited by 3
  24. 2025Pico: A Modular Framework for Hypothesis-Driven Small Language Model Researchcited by 3
  25. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 3
  26. 2026Slicing and Dicing: Configuring Optimal Mixtures of Expertscited by 2
  27. 2026Probabilistic Calibration Is a Trainable Capability in Language Modelscited by 2
  28. 2025Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Modelscited by 1
  29. 2025SLAyiNG: A Diverse and Community-validated Dataset of Queer Slangcited by 1
  30. 2026On the Emergence and Test-Time Use of Structural Information in Large Language Modelscited by 1
  31. 2026AP-OOD: Attention Pooling for Out-of-Distribution Detectioncited by 1
  32. 2026From Token Probabilities to Semantic Constraints: Towards Declarative Probabilistic Evaluation of Language Modelscited by 1
  33. 2025SASQ: Static Activation Scaling for Quantization-Aware Training in Large Language Modelscited by 0
  34. 2025Perturb Your Data: Paraphrase-Guided Training Data Watermarkingcited by 0
  35. 2026Detecting Non-Membership in LLM Training Data via Rank Correlationscited by 0
  36. 2026Fidelity-Diversity Metrics for Textcited by 0
  37. 2026Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extensioncited by 0
  38. 2026Evaluating Tiny Recursive Models Across Training for Code Generationcited by 0

50 most cited of 54 All 54 (JSON)