mlsanityData to 7 Oct 2026

Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps

arXiv 2407.07071 ↗First posted 9 Jul 2024cs.CL cs.AI cs.LGcited by 140

Not on the map. Why?

Abstract (arXiv)

When asked to summarize articles or answer questions given a passage, large language models (LLMs) can hallucinate details and respond with unsubstantiated answers that are inaccurate with respect to the input context. This paper describes a simple approach for detecting such contextual hallucinations. We hypothesize that contextual hallucinations are related to the extent to which an LLM attends to information in the provided context versus its own generations. Based on this intuition, we propose a simple hallucination detection model whose input features are given by the ratio of attention weights on the context versus newly generated tokens (for each attention head). We find that a linear classifier based on these lookback ratio features is as effective as a richer detector that utilizes the entire hidden states of an LLM or a text-based entailment model. The lookback ratio-based detector -- Lookback Lens -- is found to transfer across tasks and even models, allowing a detector that is trained on a 7B model to be applied (without retraining) to a larger 13B model. We further apply this detector to mitigate contextual hallucinations, and find that a simple classifier-guided decoding approach is able to reduce the amount of hallucination, for example by 9.6% in the XSum summarization task.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

35 of 52 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
23 more
  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  3. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  4. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  9. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  10. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  11. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  12. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  13. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  14. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  15. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  16. 2020Self-Attention Attribution: Interpreting Information Interactions Inside Transformercited by 124
  17. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  18. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  19. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
  20. 2019Attention Interpretability Across NLP Taskscited by 71
  21. 2023Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuningcited by 45
  22. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 43
  23. 2024Constructing Benchmarks and Interventions for Combating Hallucinations in LLMscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 63; 2026: 65.

140 papers.

Sort:

By area:

  1. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  2. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  3. 2024Attention Heads of Large Language Models: A Surveycited by 65
  4. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  5. 2025Hallucination Detection in LLMs Using Spectral Features of Attention Mapscited by 42
  6. 2024Probing-RAG: Self-Probing to Guide Language Models in Selective Document Retrievalcited by 28
  7. 2024DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinationscited by 23
  8. 2025A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputscited by 20
  9. 2025SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Modelscited by 19
  10. 2025GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environmentscited by 18
  11. 2025Hallucination Detection in LLMs with Topological Divergence on Attention Graphscited by 18
  12. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
38 more
  1. 2024When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated Explanationscited by 16
  2. 2025Attention-guided Self-reflection for Zero-shot Hallucination Detection in Large Language Modelscited by 16
  3. 2025Explainable Chain-of-Thought Reasoning: An Empirical Analysis on State-Aware Reasoning Dynamicscited by 16
  4. 2025FactSelfCheck: Fact-Level Black-Box Hallucination Detection for LLMscited by 14
  5. 2025Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Headscited by 14
  6. 2025Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Modelscited by 14
  7. 2025Enhancing Mathematical Reasoning in Large Language Models with Self-Consistency-Based Hallucination Detectioncited by 13
  8. 2025When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQAcited by 13
  9. 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMscited by 12
  10. 2025Learning to Attribute with Attentioncited by 11
  11. 2025Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviorscited by 11
  12. 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performancecited by 10
  13. 2025Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillationcited by 10
  14. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  15. 2025Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directionscited by 10
  16. 2024Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Modelscited by 9
  17. 2025Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Modelscited by 9
  18. 2025Shadows in the Attention: Contextual Perturbation and Representation Drift in the Dynamics of Hallucination in LLMscited by 9
  19. 2024Integrative Decoding: Improve Factuality via Implicit Self-consistencycited by 8
  20. 2024Differential Transformercited by 8
  21. 2025Language Models Can Predict Their Own Behaviorcited by 8
  22. 2025Do Sparse Autoencoders Generalize? A Case Study of Answerabilitycited by 8
  23. 2025Risk Assessment Framework for Code LLMs via Leveraging Internal Statescited by 8
  24. 2025A Single Direction of Truth: An Observer Model's Linear Residual Probe Exposes and Steers Contextual Hallucinationscited by 8
  25. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lenscited by 8
  26. 2026VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneckcited by 8
  27. 2026ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time Correctioncited by 8
  28. 2025HICD: Hallucination-Inducing via Attention Dispersion for Contrastive Decoding to Mitigate Hallucinations in Large Language Modelscited by 7
  29. 2025Towards Long Context Hallucination Detectioncited by 7
  30. 2026TDGNet: Hallucination Detection in Diffusion Language Models via Temporal Dynamic Graphscited by 7
  31. 2025Large Language Models Penetration in Scholarly Writing and Peer Reviewcited by 6
  32. 2025Hallucinated Span Detection with Multi-View Attention Featurescited by 6
  33. 2025Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarizationcited by 6
  34. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  35. 2025Uncertainty as Feature Gaps: Epistemic Uncertainty Quantification of LLMs in Contextual Question-Answeringcited by 6
  36. 2024Attention with Dependency Parsing Augmentation for Fine-Grained Attributioncited by 5
  37. 2025Improving the Reliability of LLMs: Combining CoT, RAG, Self-Consistency, and Self-Verificationcited by 5
  38. 2025Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generationcited by 5

50 most cited of 140 All 140 (JSON)