mlsanityData to 14 Oct 2026

Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings

arXiv 2505.15353 ↗First posted 21 May 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Log-likelihood vectors define a common space for comparing language models as probability distributions, enabling unified comparisons across heterogeneous settings. We extend this framework to training checkpoints and intermediate layers, and establish a consistent scale for KL divergence across pretraining, model size, random seeds, quantization, fine-tuning, and layers. Analysis of Pythia pretraining trajectories further shows that changes in log-likelihood space, as measured by the scaling behavior of KL divergence, are much smaller than in weight space, resulting in subdiffusive learning trajectories and early stabilization of language-model behavior despite weight drift.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 64 of its 105 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

64 of 105 references.

References · 64

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
38 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  13. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  14. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  15. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  16. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  17. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  18. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  19. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  20. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  21. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  22. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  23. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  24. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  25. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  26. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  27. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  28. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  29. 2018Gender Bias in Coreference Resolutioncited by 442
  30. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  31. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  32. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  33. 2020Query-Key Normalization for Transformerscited by 280
  34. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  35. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  36. 2016Benefits of depth in neural networkscited by 250
  37. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  38. 2024Generative Representational Instruction Tuningcited by 233

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Modelcited by 0
  2. 2026Language Model Maps for Prompt-Response Distributions via Log-Likelihood Vectorscited by 0
  3. 2026How Divergence Becomes Decision Flips in Compressed Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
  2. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
  3. 2025Better Estimation of the Kullback--Leibler Divergence Between Language Models
  4. 2026Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation
  5. 2024Observational Scaling Laws and the Predictability of Language Model Performance
  6. 2026Representation Collapse in Sequential Post-Training of Large Language Models
  7. 2023Likelihood-Based Diffusion Language Models
  8. 2025A Markov Categorical Framework for Language Modeling