The Alignment Bottleneck
Not on the map. Why?
Abstract (arXiv)
Large language models improve with scale, yet feedback-based alignment still exhibits systematic deviations from intended behavior. Motivated by bounded rationality in economics and cognitive science, we view judgment as resource-limited and feedback as a constrained channel. On this basis, we model the loop as a two-stage cascade U → H → Y given S, with cognitive capacity Ccog|S and average total capacity barCtot|S. Our main result is a capacity-coupled Alignment Performance Interval. It pairs a data size-independent Fano lower bound proved on a separable codebook mixture with a PAC-Bayes upper bound whose KL term is controlled by the same channel via m , barCtot|S. The PAC-Bayes bound becomes an upper bound on the same true risk when the canonical observable loss is used and the dataset is drawn from the same mixture. Under these matched conditions, both limits are governed by a single capacity. Consequences include that, with value complexity and capacity fixed, adding labels alone cannot cross the bound; attaining lower risk on more complex targets requires capacity that grows with log M; and once useful signal saturates capacity, further optimization tends to fit channel regularities, consistent with reports of sycophancy and reward hacking. The analysis views alignment as interface engineering: measure and allocate limited capacity, manage task complexity, and decide where information is spent.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 62 of its 78 references that are in the map. First posted on arXiv 19 Sep 2025.
Builds on
62 of 78 references.
References · 62
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2016Concrete Problems in AI Safetycited by 1,642
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2016Deep Variational Information Bottleneckcited by 930
38 more
- 2015Deep Learning and the Information Bottleneck Principlecited by 802
- 2017Spectrally-normalized margin bounds for neural networkscited by 686
- 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
- 2017Exploring Generalization in Deep Learningcited by 600
- 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
- 2018AI safety via debatecited by 400
- 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
- 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
- 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
- 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
- 2024A Roadmap to Pluralistic Alignmentcited by 184
- 2023Mitigating the Alignment Tax of RLHFcited by 183
- 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
- 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
- 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
- 2024Rule Based Rewards for Language Model Safetycited by 108
- 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 107
- 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 105
- 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
- 2023How Does Information Bottleneck Help Deep Learning?cited by 93
- 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 79
- 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
- 2017Nonlinear Information Bottleneckcited by 71
- 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
- 2024Axioms for AI Alignment from Human Feedbackcited by 52
- 2021People construct simplified mental representations to plancited by 47
- 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 27
- 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
- 2021Deciding What to Learn: A Rate-Distortion Approachcited by 23
- 2021User Tampering in Reinforcement Learning Recommender Systemscited by 23
- 2024Information Theoretic Guarantees For Policy Alignment In Large Language Modelscited by 23
- 2021Cognitive science as a source of forward and inverse models of human decisions for robotics and controlcited by 22
- 2020A Rate-Distortion view of human pragmatic reasoningcited by 19
- 2022Deciding What to Model: Value-Equivalent Sampling for Reinforcement Learningcited by 19
- 2023More PAC-Bayes bounds: From bounded losses, to losses with general tail behaviors, to anytime validitycited by 17
- 2021The Value of Information When Deciding What to Learncited by 15
- 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
50 most cited of 62 All 62 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Inference-time Alignment in Continuous Space
- 2025Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time
- 2026Oracle-Robust Online Alignment for Large Language Models
- 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment
- 2026Theoretical Limits of Language Model Alignment
- 2026Towards a theory of inference-time alignment with unknown rewards
- 2026Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling
- 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time