mlsanityData to 7 Oct 2026

Language Model Evaluation Beyond Perplexity

arXiv 2106.00085 ↗First posted 31 May 2021cs.CLcited by 77

Not on the map. Why?

Abstract (arXiv)

We propose an alternate approach to quantifying how well language models learn natural language: we ask how well they match the statistical tendencies of natural language. To answer this question, we analyze whether text generated from language models exhibits the statistical tendencies present in the human-generated text on which they were trained. We provide a framework--paired with significance tests--for evaluating the fit of language models to these trends. We find that neural language models appear to learn only a subset of the tendencies considered, but align much more closely with empirical trends than proposed theoretical distributions (when present). Further, the fit to different distributions is highly-dependent on both model architecture and generation strategy. As concrete examples, text generated under the nucleus sampling scheme adheres more closely to the type--token relationship of natural language than text produced using standard ancestral sampling; text from LSTMs reflects the natural language distributions over length, stopwords, and symbols surprisingly well.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 23 of its 49 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

23 of 49 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2019How Can We Know What Language Models Know?cited by 953
  7. 2016Language Modeling with Gated Convolutional Networkscited by 688
  8. 2019Designing and Interpreting Probes with Control Taskscited by 608
  9. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  10. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  11. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  12. 2018Colorless green recurrent networks dream hierarchicallycited by 210
11 more
  1. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  2. 2019Sparse Sequence-to-Sequence Modelscited by 167
  3. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  4. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  5. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  6. 2020Improved Natural Language Generation via Loss Truncationcited by 75
  7. 2018Deep RNNs Encode Soft Hierarchical Syntaxcited by 34
  8. 2019Evaluating Computational Language Models with Scaling Properties of Natural Languagecited by 20
  9. 2018A Neural Model of Adaptation in Readingcited by 18
  10. 2017Do Neural Nets Learn Statistical Laws behind Natural Language?cited by 16
  11. 2016Length bias in Encoder Decoder Models and a Case for Global Conditioningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 12; 2023: 18; 2024: 18; 2025: 14; 2026: 12.

77 papers.

Sort:

By area:

  1. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  2. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  3. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  4. 2023Ghostbuster: Detecting Text Ghostwritten by Large Language Modelscited by 90
  5. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  6. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  7. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  8. 2021Coherence boosting: When your pretrained language model is not paying enough attentioncited by 38
  9. 2023Emergent Representations of Program Semantics in Language Models Trained on Programscited by 28
  10. 2024REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answeringcited by 28
  11. 2024Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Modelscited by 24
  12. 2022LMentry: A Language Model Benchmark of Elementary Language Taskscited by 21
38 more
  1. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  2. 2022N-Best Hypotheses Reranking for Text-To-SQL Systemscited by 19
  3. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  4. 2022Evaluating Distributional Distortion in Neural Language Modelingcited by 18
  5. 2022Measuring Datacited by 18
  6. 2023Meaning Representations from Trajectories in Autoregressive Modelscited by 18
  7. 2023Semantic Compression With Large Language Modelscited by 17
  8. 2025Confounding Factors in Relating Model Performance to Morphologycited by 14
  9. 2024Predictions from language models for multiple-choice tasks are not robust under variation of scoring methodscited by 13
  10. 2022Probing BERT's priors with serial reproduction chainscited by 11
  11. 2023ChatGPT in the Age of Generative AI and Large Language Models: A Concise Surveycited by 10
  12. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 10
  13. 2024Large Models of What? Mistaking Engineering Achievements for Human Linguistic Agencycited by 9
  14. 2024GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Modelscited by 9
  15. 2024The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generationcited by 9
  16. 2023CPopQA: Ranking Cultural Concept Popularity by LLMscited by 8
  17. 2024The GPT-WritingPrompts Dataset: A Comparative Analysis of Character Portrayal in Short Storiescited by 8
  18. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 8
  19. 2021Modeling the Unigram Distributioncited by 7
  20. 2022On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluationcited by 7
  21. 2023BrainLLM: Generative Language Decoding from Brain Recordingscited by 7
  22. 2022Link the World: Improving Open-domain Conversation with Dynamic Spatiotemporal-aware Knowledgecited by 6
  23. 2022A Natural Bias for Language Generation Modelscited by 6
  24. 2023Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Featurescited by 6
  25. 2023LEAP: Efficient and Automated Test Method for NLP Softwarecited by 6
  26. 2024Prompting Large Language Models with Audio for General-Purpose Speech Summarizationcited by 6
  27. 2024On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruningcited by 6
  28. 2022Predictive Querying for Autoregressive Neural Sequence Modelscited by 5
  29. 2025On the Empirical Power of Goodness-of-Fit Tests in Watermark Detectioncited by 5
  30. 2025PRISM: Enhancing Protein Inverse Folding through Fine-Grained Retrieval on Structure-Sequence Multimodal Representationscited by 5
  31. 2024Predict the Next Word: Humans exhibit uncertainty in this task and language models _____cited by 4
  32. 2024Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Modelscited by 4
  33. 2025How Sampling Affects the Detectability of Machine-written texts: A Comprehensive Studycited by 4
  34. 2024TwinBooster: Synergising Large Language Models with Barlow Twins and Gradient Boosting for Enhanced Molecular Property Predictioncited by 3
  35. 2024Language Modeling Using Tensor Trainscited by 3
  36. 2025Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priorscited by 3
  37. 2023Learning from Red Teaming: Gender Bias Provocation and Mitigation in Large Language Modelscited by 2
  38. 2024Thread Detection and Response Generation using Transformers with Prompt Optimisationcited by 2

50 most cited of 77 All 77 (JSON)