Human Supervision as an Information Bottleneck: A Unified Theory of Error Floors in Human-Guided Learning
Not on the map. Why?
Abstract (arXiv)
Large language models are trained primarily on human-generated data and feedback, yet they exhibit persistent errors arising from annotation noise, subjective preferences, and the limited expressive bandwidth of natural language. We argue that these limitations reflect structural properties of the supervision channel rather than model scale or optimization. We develop a unified theory showing that whenever the human supervision channel is not sufficient for a latent evaluation target, it acts as an information-reducing channel that induces a strictly positive excess-risk floor for any learner dominated by it. We formalize this Human-Bounded Intelligence limit and show that across six complementary frameworks (operator theory, PAC-Bayes, information theory, causal inference, category theory, and game-theoretic analyses of reinforcement learning from human feedback), non-sufficiency yields strictly positive lower bounds arising from the same structural decomposition into annotation noise, preference distortion, and semantic compression. The theory explains why scaling alone cannot eliminate persistent human-aligned errors and characterizes conditions under which auxiliary non-human signals (e.g., retrieval, program execution, tools) increase effective supervision capacity and collapse the floor by restoring information about the latent target. Experiments on real preference data, synthetic known-target tasks, and externally verifiable benchmarks confirm the predicted structural signatures: human-only supervision exhibits a persistent floor, while sufficiently informative auxiliary channels strictly reduce or eliminate excess error.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 26 references that are in the map. First posted on arXiv 26 Feb 2026.
Builds on
15 of 26 references.
References · 15
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
3 more
- 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
- 2022Defining and Characterizing Reward Hackingcited by 203
- 2019A Primer on PAC-Bayesian Learningcited by 132
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback
- 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
- 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
- 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuning
- 2026Why Code, Why Now: An Information-Theoretic Perspective on the Limits of Machine Learning
- 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
- 2026Thinking Outside the Box: Can Language Models Rely on External Guidance Selectively?
- 2023SuperHF: Supervised Iterative Learning from Human Feedback