mlsanityData to 7 Oct 2026

The Behavioral Credibility Trilemma: When Calibrated Autonomy Becomes Impossible

arXiv 2605.25739 ↗First posted 25 May 2026cs.LG cs.GT stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

We prove that no reinforcement learning policy with confidence-gated autonomy can simultaneously achieve maximum helpfulness, optimal calibration, and full autonomy under rational oversight, whenever some tasks exceed the agent's reliable competence: the Behavioral Credibility Trilemma. The impossibility is geometric: adding any non-affine autonomy incentive to a strictly proper scoring rule destroys strict properness, so an agent rewarded for both calibrated confidence and autonomous action systematically inflates its reported confidence on tasks below the principal's approval threshold whenever the autonomy stake exceeds the calibration cost of clearing it. The Behavioral Perturbation Lemma quantifies the inflation (scaling as wA/(2 wC) for the Brier score) and shows detection requires Omega(1/Delta2) observations for interior reports. We prove that, in the unsaturated regime, no affine oversight rule is optimal for the principal and the optimum is attained by a sharp threshold satisfying the trilemma's own hypotheses, so the impossibility is endogenized rather than assumed; moreover, for symmetric, log-concave, full-support location policy families under the Brier score, calibration is not even a stationary point of policy-gradient training. We formalize the Confidence-Gated Decision Problem, map existing methods onto the trilemma, and identify two constructive resolution pathways (commitment, role separation). A 540-configuration Best-of-N experiment tests five hypotheses, all strongly confirmed (effect sizes d = 1.10 to 5.35, the upper end from a per-completion estimator inflating magnitude over per-task aggregates) and replicated under a pre-specified protocol on two further model families, and adds a descriptive analysis of the achievable-(H, C, A) surface geometry showing a plateau-truncated frontier consistent with the predicted inflation saturation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 107 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

41 of 107 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2019Advances and Open Problems in Federated Learningcited by 3,021
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
29 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  3. 2017Constrained Policy Optimizationcited by 928
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  9. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  10. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  11. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  12. 2022Self-critiquing models for assisting human evaluatorscited by 372
  13. 2020Performative Predictioncited by 318
  14. 2020Calibration of Pre-trained Transformerscited by 307
  15. 2015Strategic Classificationcited by 256
  16. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  17. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  18. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  19. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  20. 2018Categorizing Variants of Goodhart's Lawcited by 132
  21. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  22. 2018Incomplete Contracting and AI Alignmentcited by 60
  23. 2024When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedbackcited by 17
  24. 2026Sycophantic AI makes human interaction feel more effortful and less satisfying over timecited by 11
  25. 2021Incentive-Compatible Forecasting Competitionscited by 9
  26. 2023Incentivizing honest performative predictions with proper scoring rulescited by 9
  27. 2025Murphys Laws of AI Alignment: Why the Gap Always Winscited by 4
  28. 2026Honest Reporting in Scored Oversight: True-KL0 Property via the Prekopa Principlecited by 2
  29. 2026The Endogeneity of Miscalibration: Impossibility and Escape in Scored Reportingcited by 1

Built on by

Not cited yet.