mlsanityData to 14 Oct 2026

Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging

arXiv 2510.17426 ↗First posted 20 Oct 2025cs.CL cs.AI cs.LGcited by 10

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The "alignment tax" of post-training is typically framed as a drop in task accuracy. We show it also involves a severe loss of calibration, making models overconfident, less reliable, and model outputs less diverse. We show that this trade-off can be navigated effectively via a simple post-hoc intervention: interpolating between a model's weights before and after alignment. Crucially, this is not a strict trade-off. We find that the process consistently reveals Pareto-optimal interpolations - models that improve accuracy beyond both parents while substantially recovering the calibration lost during alignment. Our work demonstrates that simple model merging provides a computationally efficient method for mitigating the full scope of the alignment tax, yielding models that are more capable and more reliable.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 45 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

28 of 45 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment28 May
  2. 2024Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction22 May
  3. 2023Mitigating the Alignment Tax of RLHF12 Sep
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  11. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  12. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
12 more
  1. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  2. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  3. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  4. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  5. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  6. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  7. 2023Generative Language Models Exhibit Social Identity Biasescited by 111
  8. 2024What Large Language Models Know and What People Think They Knowcited by 110
  9. 2024DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Samplingcited by 81
  10. 2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
  11. 2025Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Modelscited by 10
  12. 2024If You Can't Use Them, Recycle Them: Optimizing Merging at Scale Mitigates Performance Tradeoffscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Demystifying Multi-Agent Debate: The Role of Confidence and Diversitycited by 18
  2. 2026Confidence Estimation for LLMs in Multi-turn Interactionscited by 12
  3. 2026Value of Information: A Framework for Human-Agent Communicationcited by 12
  4. 2026Steer Model beyond Assistant: Controlling System Prompt Strength via Contrastive Decodingcited by 2
  5. 2026EpiCaR: Knowing What You Don't Know Matters for Better Reasoning in LLMscited by 2
  6. 2026Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answercited by 2
  7. 2026Not All Layers Need Tuning: Selective Layer Restoration Recovers Diversitycited by 1
  8. 2026What Is the Alignment Tax?cited by 1
  9. 2026Identifying Influential N-grams in Confidence Calibration via Regression Analysiscited by 0
  10. 2026OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System Onecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blending
  2. 2026Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMs
  3. 2025From Coefficients to Directions: Rethinking Model Merging with Directional Alignment
  4. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models
  5. 2026Consistency Training Can Entrench Misalignment
  6. 2026Uncovering Cross-Objective Interference in Multi-Objective Alignment
  7. 2026Overtrained, Not Misaligned
  8. 2025Model Merging via Multi-Teacher Knowledge Distillation