mlsanityData to 14 Oct 2026

Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding

arXiv 2606.21906 ↗First posted 20 Jun 2026cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Autoregressive generation in large language models (LLMs) conventionally decodes from the final layer, assuming that deeper representations yield more reliable next-token predictions. We revisit this assumption by revealing a recurring Guess-Refine-Perturb dynamic: early layers form coarse guesses, intermediate layers refine reasoning-relevant semantics, and final layers can perturb these refined predictions toward generic or alignment-preferred tokens. We introduce Confident Decoding, a training-free decoding strategy that dynamically selects the most reliable near-final layer through entropy-guided conservative backward search. We further provide a theoretical formulation of layer selection as an optimal stopping problem, showing that under bounded projection noise and dominant late-stage alignment perturbation, our search rule filters perturbation while bounding the loss relative to the oracle refinement layer. Experiments across dense and Mixture-of-Experts LLMs demonstrate consistent gains on challenging reasoning benchmarks, including GPQA-Diamond, Omni-MATH, and HLE, with zero memory overhead and less than 2% latency increase. These results suggest dynamically bypassing final-layer perturbations can unlock stronger reasoning behavior from aligned LLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 56 references that are in the map. First posted on arXiv 20 Jun 2026.

Builds on

52 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Mitigating the Alignment Tax of RLHF12 Sep
  2. 2022Training language models to follow instructions with human feedback4 Mar
  3. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
37 more
  1. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  2. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  3. 2018Universal Transformerscited by 647
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  6. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  7. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  8. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  10. 2025Humanity's Last Examcited by 491
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  12. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  13. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  14. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  15. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  16. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  17. 2022Confident Adaptive Language Modelingcited by 310
  18. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  19. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  20. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  21. 2025Dynamic Early Exit in Reasoning Modelscited by 219
  22. 2024Chain-of-Thought Reasoning Without Promptingcited by 210
  23. 2023Continual Pre-training of Language Modelscited by 198
  24. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 106
  25. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  26. 2025Do Language Models Use Their Depth Efficiently?cited by 50
  27. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 32
  28. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 27
  29. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Modelscited by 16
  30. 2025How Do LLMs Use Their Depth?cited by 16
  31. 2025Active Layer-Contrastive Decoding Reduces Hallucination in Large Language Model Generationcited by 10
  32. 2024Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Modelscited by 4
  33. 2025LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layerscited by 3
  34. 2025Cognition-of-Thought Elicits Social-Aligned Reasoning in Large Language Modelscited by 3
  35. 2025Generalization or Memorization: Dynamic Decoding for Mode Steeringcited by 3
  36. 2026Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Modelscited by 3
  37. 2026The Diminishing Returns of Early-Exit Decoding in Modern LLMscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Write Back the Delta: Revisiting the Same Tokens with Fresh Representationscited by 0
  2. 2026LatentHarness: Learning Latent Actions for Memory and Reasoning via Counterfactual Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Think Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning
  2. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
  3. 2026The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models
  4. 2024DeAL: Decoding-time Alignment for Large Language Models
  5. 2026Thinking Before Constraining: A Unified Decoding Framework for Large Language Models
  6. 2026Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Time
  7. 2026Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability
  8. 2025ENIGMA: The Geometry of Reasoning and Alignment in Large-Language Models