mlsanityData to 14 Oct 2026

Attention-Guided Layer Selection for Contrastive Decoding in Large Language Models

arXiv 2607.23067 ↗First posted 25 Jul 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Contrastive decoding methods such as DoLa improve the factuality of Large Language Models (LLMs) by contrasting the output distributions of mature and premature layers. However, DoLa's dynamic layer selection relies solely on divergences in output vocabulary distributions. In this work, we propose three attention-guided strategies: Attention-JSD, Attention-Entropy-Max, and Attention-Entropy-Min, which leverage structural information carried by internal self-attention mechanisms as a signal for layer selection. Experimental results on TruthfulQA demonstrate that our strategies, particularly Attention-JSD and Attention-Entropy-Min, consistently outperform the original DoLa. We observe significant gains on multi-answer metrics (MC2 and MC3), suggesting that attention distributions can provide a more sensitive signal for resolving factual knowledge than output vocabulary distributions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 25 Jul 2026.

Builds on

16 of 23 references.

References · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  9. 2022Mass-Editing Memory in a Transformercited by 1,001
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  11. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  12. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
4 more
  1. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  2. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 142
  3. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 105
  4. 2024Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contextscited by 19

Built on by

Not cited yet.