mlsanityData to 14 Oct 2026

CATTO: Balancing Preferences and Confidence in Language Models

arXiv 2601.23096 ↗First posted 30 Jan 2026cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models (LLMs) often make accurate next token predictions but their confidence in these predictions can be poorly calibrated: high-confidence predictions are frequently wrong, and low-confidence predictions may be correct. This miscalibration is exacerbated by preference-based alignment methods breaking the link between predictive probability and correctness. We introduce a Calibration Aware Token-level Training Objective (CATTO), a calibration-aware objective that aligns predicted confidence with empirical prediction correctness, which can be combined with the original preference optimization objectives. Empirically, CATTO reduces Expected Calibration Error (ECE) by 2.22%-7.61% in-distribution and 1.46%-10.44% out-of-distribution compared to direct preference optimization (DPO), and by 0.22%-1.24% in-distribution and 1.23%-5.07% out-of-distribution compared to the strongest DPO baseline. This improvement in confidence does not come at a cost of losing task accuracy, where CATTO maintains or slightly improves multiple-choice question-answering accuracy on five datasets. We also introduce Confidence@k, a test-time scaling mechanism leveraging calibrated token probabilities for Bayes-optimal selection of output tokens.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 46 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

29 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach4 May
  2. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  3. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization26 May
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  12. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
13 more
  1. 2019Measuring Calibration in Deep Learningcited by 446
  2. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
  3. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  4. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  5. 2018Stochastic subgradient method converges on tame functionscited by 124
  6. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  7. 2021Soft Calibration Objectives for Neural Networkscited by 92
  8. 2023On the Calibration of Large Language Models and Alignmentcited by 73
  9. 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 72
  10. 2021Local Calibration: Metrics and Recalibrationcited by 24
  11. 2024Confidence Calibration of Classifiers with Many Classescited by 11
  12. 2025LookAlike: Consistent Distractor Generation in Math MCQscited by 7
  13. 2025Calibrated Language Models and How to Find Them with Label Smoothingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustnesscited by 4
  2. 2026Neither Parallel Nor Sequential: How DiffusionGemma Actually Commits Tokenscited by 3
  3. 2026Knowing When Not to Answer: Abstention-Aware Scientific Reasoningcited by 2
  4. 2026Quantisation Reshapes the Metacognitive Geometry of Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs
  2. 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
  3. 2026Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Models
  4. 2025Direct Confidence Alignment: Aligning Verbalized Confidence with Internal Confidence In Large Language Models
  5. 2026ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
  6. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  7. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  8. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models