mlsanityData to 14 Oct 2026

Non-Vacuous Generalization Bounds for Large Language Models

arXiv 2312.17173 ↗First posted 28 Dec 2023stat.ML cs.LGcited by 53

Not on the map. Why?

Abstract (arXiv)

Modern language models can contain billions of parameters, raising the question of whether they can generalize beyond the training data or simply parrot their training corpora. We provide the first non-vacuous generalization bounds for pretrained large language models (LLMs), indicating that language models are capable of discovering regularities that generalize to unseen data. In particular, we derive a compression bound that is valid for the unbounded log-likelihood loss using prediction smoothing, and we extend the bound to handle subsampling, accelerating bound computation by orders of magnitude on massive datasets. To achieve the extreme level of compression required for non-vacuous bounds, we devise SubLoRA, a simple low-dimensional nonlinear parameterization that leads to non-vacuous generalization bounds for models with nearly a billion parameters. Finally, we use our bounds to understand LLM generalization and find that larger models have better generalization bounds and are more compressible than smaller models.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

33 of 48 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2023Mistral 7Bcited by 3,164
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  12. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
21 more
  1. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  2. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  3. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  4. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  5. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  6. 2017Implicit Regularization in Matrix Factorizationcited by 322
  7. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  8. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  9. 2023Language Modeling Is Compressioncited by 225
  10. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
  11. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 126
  12. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  13. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  14. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 100
  15. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 88
  16. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 76
  17. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  18. 2020PAC-Bayes unleashed: generalisation bounds with unbounded lossescited by 43
  19. 2019PAC-Bayes under potentially heavy tailscited by 16
  20. 2019Efron-Stein PAC-Bayesian Inequalitiescited by 16
  21. 2021Causal Forecasting:Generalization Bounds for Autoregressive Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 20; 2026: 19.

53 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  2. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  3. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  4. 2025Deep Learning is Not So Mysterious or Differentcited by 56
  5. 2024Large Language Models as Markov Chainscited by 35
  6. 2024Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Modelscited by 25
  7. 2024The Complexity Dynamics of Grokkingcited by 22
  8. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 21
  9. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  10. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  11. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 11
  12. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
38 more
  1. 2025DoubleGen: Debiased Generative Modeling of Counterfactualscited by 9
  2. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  3. 2025Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformerscited by 6
  4. 2025Compute-Optimal LLMs Provably Generalize Better With Scalecited by 5
  5. 2024Sample Compression Unleashed: New Generalization Bounds for Real Valued Lossescited by 4
  6. 2024Just How Flexible are Neural Networks in Practice?cited by 3
  7. 2024Generalization Bounds via Meta-Learned Model Representations: PAC-Bayes and Sample Compression Hypernetworkscited by 3
  8. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
  9. 2025Prompts Generalize with Low Data: Non-vacuous Generalization Bounds for Optimizing Prompts with More Informative Priorscited by 3
  10. 2026Effective Reasoning Chains Reduce Intrinsic Dimensionalitycited by 3
  11. 2026Position: agentic AI orchestration should be Bayes-consistentcited by 3
  12. 2026FragileFlow: Spectral Control of Correct-but-Fragile Predictions for Foundation Model Robustnesscited by 3
  13. 2025Non-vacuous Generalization Bounds for Deep Neural Networks without any modification to the trained modelscited by 2
  14. 2025Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learningcited by 2
  15. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  16. 2025Uncertainty Quantification and Data Efficiency in AI: An Information-Theoretic Perspectivecited by 2
  17. 2026Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGDcited by 2
  18. 2026Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testingcited by 2
  19. 2026Bound to Disagree: Generalization Bounds via Certifiable Surrogatescited by 2
  20. 2024A practical generalization metric for deep networks benchmarkingcited by 1
  21. 2024ViDAS: Vision-based Danger Assessment and Scoringcited by 1
  22. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  23. 2025Towards Mitigation of Hallucination for LLM-empowered Agents: Progressive Generalization Bound Exploration and Watchdog Monitorcited by 1
  24. 2025The Alignment Bottleneckcited by 1
  25. 2025Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixturescited by 1
  26. 2026Golden Handcuffs make safer AI agentscited by 1
  27. 2026Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Datacited by 1
  28. 2026Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewardscited by 1
  29. 2025Model Diffusion for Certifiable Few-shot Transfer Learningcited by 0
  30. 2025`Generalization is hallucination' through the lens of tensor completionscited by 0
  31. 2025Customizing the Inductive Biases of Softmax Attention using Structured Matricescited by 0
  32. 2026Sparse Semantic Dimension as a Generalization Certificate for LLMscited by 0
  33. 2026Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimizationcited by 0
  34. 2026PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theorycited by 0
  35. 2026From Privacy to Generalization: Linear Max-Information Bounds for Differentially Private Learning Algorithmscited by 0
  36. 2026Inductive Generalization for Robotic Manipulationcited by 0
  37. 2026Understanding Large Language Modelscited by 0
  38. 2026Certification Frontiers for Gaussian LoRA: Independent Priors, Posterior Risk, and Prediction-Preserving Balancingcited by 0

50 most cited of 53 All 53 (JSON)