mlsanityData to 14 Oct 2026

The Alignment Bottleneck

arXiv 2509.15932 ↗First posted 19 Sep 2025cs.LG cs.AI cs.ITcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models improve with scale, yet feedback-based alignment still exhibits systematic deviations from intended behavior. Motivated by bounded rationality in economics and cognitive science, we view judgment as resource-limited and feedback as a constrained channel. On this basis, we model the loop as a two-stage cascade U → H → Y given S, with cognitive capacity Ccog|S and average total capacity barCtot|S. Our main result is a capacity-coupled Alignment Performance Interval. It pairs a data size-independent Fano lower bound proved on a separable codebook mixture with a PAC-Bayes upper bound whose KL term is controlled by the same channel via m , barCtot|S. The PAC-Bayes bound becomes an upper bound on the same true risk when the canonical observable loss is used and the dataset is drawn from the same mixture. Under these matched conditions, both limits are governed by a single capacity. Consequences include that, with value complexity and capacity fixed, adding labels alone cannot cross the bound; attaining lower risk on more complex targets requires capacity that grows with log M; and once useful signal saturates capacity, further optimization tends to fit channel regularities, consistent with reports of sycophancy and reward hacking. The analysis views alignment as interface engineering: measure and allocate limited capacity, manage task complexity, and decide where information is spent.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 62 of its 78 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

62 of 78 references.

References · 62

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2016Concrete Problems in AI Safetycited by 1,642
  11. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  12. 2016Deep Variational Information Bottleneckcited by 930
38 more
  1. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  2. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  4. 2017Exploring Generalization in Deep Learningcited by 600
  5. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  6. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  7. 2018AI safety via debatecited by 400
  8. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  9. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  10. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  11. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  12. 2024A Roadmap to Pluralistic Alignmentcited by 184
  13. 2023Mitigating the Alignment Tax of RLHFcited by 183
  14. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  15. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  16. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  17. 2024Rule Based Rewards for Language Model Safetycited by 108
  18. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 107
  19. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 105
  20. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  21. 2023How Does Information Bottleneck Help Deep Learning?cited by 93
  22. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 79
  23. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  24. 2017Nonlinear Information Bottleneckcited by 71
  25. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  26. 2024Axioms for AI Alignment from Human Feedbackcited by 52
  27. 2021People construct simplified mental representations to plancited by 47
  28. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 27
  29. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  30. 2021Deciding What to Learn: A Rate-Distortion Approachcited by 23
  31. 2021User Tampering in Reinforcement Learning Recommender Systemscited by 23
  32. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Modelscited by 23
  33. 2021Cognitive science as a source of forward and inverse models of human decisions for robotics and controlcited by 22
  34. 2020A Rate-Distortion view of human pragmatic reasoningcited by 19
  35. 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
  36. 2023More PAC-Bayes bounds: From bounded losses, to losses with general tail behaviors, to anytime validitycited by 17
  37. 2021The Value of Information When Deciding What to Learncited by 15
  38. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025Inference-time Alignment in Continuous Space
  2. 2025Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time
  3. 2026Oracle-Robust Online Alignment for Large Language Models
  4. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment
  5. 2026Theoretical Limits of Language Model Alignment
  6. 2026Towards a theory of inference-time alignment with unknown rewards
  7. 2026Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling
  8. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time