mlsanityData to 14 Oct 2026

Attention Consistency for LLMs Explanation

arXiv 2509.17178 ↗First posted 21 Sep 2025cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Understanding the decision-making processes of large language models (LLMs) is essential for their trustworthy development and deployment. However, current interpretability methods often face challenges such as low resolution and high computational cost. To address these limitations, we propose the Multi-Layer Attention Consistency Score (MACS), a novel, lightweight, and easily deployable heuristic for estimating the importance of input tokens in decoder-based models. MACS measures contributions of input tokens based on the consistency of maximal attention. Empirical evaluations demonstrate that MACS achieves a favorable trade-off between interpretability quality and computational efficiency, showing faithfulness comparable to complex techniques with a 22% decrease in VRAM usage and 30% reduction in latency.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 21 Sep 2025.

Builds on

23 of 34 references.

References · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  7. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  8. 2015Teaching Machines to Read and Comprehendcited by 1,049
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  10. 2020Quantifying Attention Flow in Transformerscited by 775
  11. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  12. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 199
11 more
  1. 2018Local Rule-Based Explanations of Black Box Decision Systemscited by 166
  2. 2025Why do LLMs attend to the first token?cited by 127
  3. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 120
  4. 2024Transformers need glasses! Information over-squashing in language taskscited by 91
  5. 2023AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulationcited by 31
  6. 2019Adding Interpretable Attention to Neural Translation Models Improves Word Alignmentcited by 30
  7. 2024Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarkscited by 28
  8. 2025Enhancing Low-Cost Video Editing with Lightweight Adaptors and Temporal-Aware Inversioncited by 28
  9. 2023Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspectivecited by 23
  10. 2025Bayesian Optimization for Controlled Image Editing via LLMscited by 11
  11. 2024Scene-Aware Explainable Multimodal Trajectory Predictioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarkscited by 15
  2. 2026Dual-Pathway Circuits of Object Hallucination in Vision-Language Modelscited by 7
  3. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
  4. 2026LLM Explainability with Counterfactual Chains and Causal Graphscited by 2
  5. 2026Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Modelscited by 1
  6. 2026BCL: Bayesian In-Context Learning Framework for Information Extractioncited by 1
  7. 2026Auditing the Audit: Five Failure Modes in Benchmark-Validity Auditscited by 1
  8. 2026E2CAR: An Efficient 2D-CNN Framework for Real-Time EEG Artifact Removal on Edge Devicescited by 0
  9. 2026Self-Correction Can Amplify Hallucinations: Fact-Level Repair with Graph-Based Evidence Routing in Multimodal Generationcited by 0
  10. 2026Probe Choice Changes Canary-Memorization Verdicts: Three Post-Hoc Disagreement Case Studies in a Text-Dominant LoRA-Tuned Autoregressive Testbedcited by 0
  11. 2026RIFT: Token Influence Follows the Path of Least Resistancecited by 0