mlsanityData to 14 Oct 2026

A Measure-Theoretic Characterization of Tight Language Models

arXiv 2212.10502 ↗First posted 20 Dec 2022cs.CL cs.FLcited by 33

Not on the map. Why?

Abstract (arXiv)

Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings. In most cases, the estimated distribution sums to 1 over all finite strings. However, in some pathological cases, probability mass can ``leak'' onto the set of infinite sequences. In order to characterize the notion of leakage more precisely, this paper offers a measure-theoretic treatment of language modeling. We prove that many popular language model families are in fact tight, meaning that they will not leak in this sense. We also generalize characterizations of tightness proposed in previous works.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 16 of its 51 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

16 of 51 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2018Deep contextualized word representationscited by 2,290
  5. 2019Language Models as Knowledge Bases?cited by 1,936
  6. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  7. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  8. 2019Sparse Sequence-to-Sequence Modelscited by 170
  9. 2022Locally Typical Samplingcited by 111
  10. 2021A Recipe For Arbitrary Text Style Transfer with Large Language Modelscited by 107
  11. 2021Revisiting the Uniform Information Density Hypothesiscited by 68
  12. 2020Limitations of Autoregressive Models and Their Alternativescited by 55
4 more
  1. 2020Consistency of a Recurrent Language Model With Respect to Incomplete Decodingcited by 54
  2. 2022Autoregressive Structured Prediction with Language Modelscited by 43
  3. 2017Recurrent Neural Networks as Weighted Language Recognizerscited by 30
  4. 2021Reconciling the Discrete-Continuous Divide: Towards a Mathematical Theory of Sparse Communicationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 14; 2025: 4; 2026: 4.

33 papers.

Sort:

By area:

  1. 2024The pitfalls of next-token predictioncited by 160
  2. 2024How to Compute the Probability of a Wordcited by 36
  3. 2024Transformers Can Represent n-gram Language Modelscited by 32
  4. 2023A Geometric Notion of Causal Probingcited by 25
  5. 2024Training Neural Networks as Recognizers of Formal Languagescited by 24
  6. 2024On the Role of Context in Reading Time Predictioncited by 20
  7. 2023A Cross-Linguistic Pressure for Uniform Information Density in Word Ordercited by 17
  8. 2023On the Efficacy of Sampling Adapterscited by 17
  9. 2023MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropiescited by 16
  10. 2025Causal Estimation of Tokenisation Biascited by 16
  11. 2024Gumbel Counterfactual Generation From Language Modelscited by 14
  12. 2024Next-token prediction capacity: general upper bounds and a lower bound for transformerscited by 11
21 more
  1. 2023Language Models as a Service: Overview of a New Paradigm and its Challengescited by 10
  2. 2024Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Taskscited by 10
  3. 2023Deriving Language Models from Masked Language Modelscited by 9
  4. 2023Recurrent Neural Language Models as Probabilistic Finite-state Automatacited by 6
  5. 2025Circuit Stability Characterizes Language Model Generalizationcited by 6
  6. 2023Structured Voronoi Samplingcited by 4
  7. 2023Principled Gradient-based Markov Chain Monte Carlo for Text Generationcited by 4
  8. 2024Lower Bounds on the Expressivity of Recurrent Neural Language Modelscited by 4
  9. 2024On Affine Homotopy between Language Encoderscited by 3
  10. 2024CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Modelscited by 3
  11. 2023On the Representational Capacity of Recurrent Neural Language Modelscited by 1
  12. 2024On Efficiently Representing Regular Languages as RNNscited by 1
  13. 2024Analyzing constrained LLM through PDFA-learningcited by 1
  14. 2025Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobscited by 1
  15. 2023Structure-Aware Path Inference for Neural Finite State Transducerscited by 0
  16. 2024A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptorscited by 0
  17. 2025A fine-grained look at causal effects in causal spacescited by 0
  18. 2026Surprisal Theory is Tautological (without Rational Grounding)cited by 0
  19. 2026The Probabilistic Structure of Large Language Modelscited by 0
  20. 2026TULIP: Targeted LLM Unlearning at Layers Identified Per-Inputcited by 0
  21. 2026Modeling the Arrival Time of Information-Bearing Events in Natural Languagecited by 0