Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time
On the map: LLM safety and security → Prompt injection and agent security
Abstract (arXiv)
We propose an influence score to quantify the contribution of attention heads to classification decisions in Transformer-based models designed for prompt injection detection. The score combines directional influence on the logits with structural contribution within the residual stream, enabling a multi-scale analysis at the head, layer, and network levels. Applied to a DeBERTa model specialized for prompt injection detection, our framework reveals distinct decision behaviours between correct and erroneous predictions. Our method provides an effective compromise between fine-grained circuit analysis and global output-based methods, and offers a systematic way to study decision mechanisms in Transformer classifiers.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 3 of its 7 references that are in the map. First posted on arXiv 4 Sep 2026.
Builds on
3 of 7 references.
References · 3
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers
- 2026From Correlation to Cause: A Five-Stage Methodology for Feature Analysis in Transformer Language Models
- 2026Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
- 2023Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers
- 2025Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers
- 2025Extracting Rule-based Descriptions of Attention Features in Transformers
- 2026Interpreting Transformers Through Attention Head Intervention
- 2025Intrinsically Interpretable Attention via Sparse Post-Training