mlsanityData to 7 Oct 2026

The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation

arXiv 2603.24124 ↗First posted 25 Mar 2026cs.LG cs.AI cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

RLHF-aligned language models exhibit response homogenization: on TruthfulQA (n=790), 40-79% of questions produce a single semantic cluster across 10 i.i.d. samples. On affected questions, sampling-based uncertainty methods have zero discriminative power (AUROC=0.500), while free token entropy retains signal (0.603). This alignment tax is task-dependent: on GSM8K (n=500), token entropy achieves 0.724 (Cohen's d=0.81). A base-vs-instruct ablation confirms the causal role of alignment: the base model shows 1.0% single-cluster rate vs. 28.5% for the instruct model (p < 10^{-6}). A training stage ablation (Base 0.0% -> SFT 1.5% -> DPO 4.0% SCR) localizes the cause to DPO, not SFT. Cross-family replication on four model families reveals alignment tax severity varies by family and scale. We validate across 22 experiments, 5 benchmarks, 4 model families, and 3 model scales (3B-14B), with Jaccard, embedding, and NLI-based baselines at three DeBERTa scales (all ~0.51 AUROC). Cross-embedder validation with two independent embedding families rules out coupling bias. Cross-dataset validation on WebQuestions (58.0% SCR) confirms generalization beyond TruthfulQA. The central finding -- response homogenization -- is implementation-independent and label-free. Motivated by this diagnosis, we explore a cheapest-first cascade (UCBD) over orthogonal uncertainty signals. Selective prediction raises GSM8K accuracy from 84.4% to 93.2% at 50% coverage; weakly dependent boundaries (|r| <= 0.12) enable 57% cost savings.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 50 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

41 of 50 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Diverse Preference Learning for Capabilities and Alignment29 Oct
  2. 2025Diverse Preference Optimization30 Jan
  3. 2024Entropy Controllable Direct Preference Optimization12 Nov
  4. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks23 Apr
  5. 2023Mitigating the Alignment Tax of RLHF12 Sep
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 35

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2016Complex Embeddings for Simple Link Predictioncited by 897
  7. 2017Selective Classification for Deep Neural Networkscited by 799
  8. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  9. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  10. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  11. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  12. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
23 more
  1. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  2. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  3. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 100
  4. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  5. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  6. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  7. 2025Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistencycited by 30
  8. 2024Metacognitive Retrieval-Augmented Large Language Modelscited by 26
  9. 2025Estimating LLM Uncertainty with Evidencecited by 26
  10. 2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
  11. 2025Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Modelscited by 18
  12. 2025Semantic Energy: Detecting LLM Hallucination Beyond Entropycited by 18
  13. 2025SINdex: Semantic INconsistency Index for Hallucination Detection in LLMscited by 14
  14. 2025Query-Level Uncertainty in Large Language Modelscited by 12
  15. 2025HalluShift: Measuring Distribution Shifts towards Hallucination Detection in LLMscited by 8
  16. 2026HALT: Hallucination Assessment via Log-probs as Time seriescited by 8
  17. 2025Uncertainty Under the Curve: A Sequence-Level Entropy Area Metric for Reasoning LLMcited by 5
  18. 2025Probabilities Are All You Need: A Probability-Only Approach to Uncertainty Estimation in Large Language Modelscited by 5
  19. 2026Semantic Uncertainty Quantification of Hallucinations in LLMs: A Quantum Tensor Network Based Methodcited by 4
  20. 2026Semantic Self-Distillation for Language Model Uncertaintycited by 4
  21. 2025Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Modelscited by 3
  22. 2025DSA, AIA, and LLMs: Approaches to conceptualizing and auditing moderation in LLM-based chatbots across languages and interfaces in the electoral contextscited by 2
  23. 2026What Is the Alignment Tax?cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026The One-Word Census: Answer-Choice Conformity Across 44 Language Modelscited by 3
  2. 2026Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)cited by 1
  3. 2026Broken Symmetry in LLM Refusal: Answer Release Is More Local Than Refusal Restorationcited by 0
  4. 2026Off-Target Effects of Response-Style Alignment in a Korean 27B Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting
  2. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignment
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  4. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  5. 2025An Uncertainty-Driven Adaptive Self-Alignment Framework for Large Language Models
  6. 2026SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment
  7. 2026The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models
  8. 2024Aligner: Efficient Alignment by Learning to Correct