# Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layers

arXiv: https://arxiv.org/abs/2409.03621 · HTML: https://mlsanity.com/paper/2409.03621/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Llemma: An Open Language Model For Mathematics](https://mlsanity.com/paper/2310.10631/) (2023)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://mlsanity.com/paper/2404.16710/) (2024)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [The Right Tool for the Job: Matching Model and Instance Complexities](https://mlsanity.com/paper/2004.07453/) (2020)
- [How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations](https://mlsanity.com/paper/1909.04925/) (2019)
- [Improving Transformer Models by Reordering their Sublayers](https://mlsanity.com/paper/1911.03864/) (2019)
- [Pay Attention when Required](https://mlsanity.com/paper/2009.04534/) (2020)

## Built on by (most cited first, 25 of 25)

- [From Tokens to Words: On the Inner Lexicon of LLMs](https://mlsanity.com/paper/2410.05864/) (2024)
- [Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance](https://mlsanity.com/paper/2411.14279/) (2024)
- [Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering](https://mlsanity.com/paper/2505.12025/) (2025)
- [AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10720/) (2025)
- [Empowering GraphRAG with Knowledge Filtering and Integration](https://mlsanity.com/paper/2503.13804/) (2025)
- [Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding](https://mlsanity.com/paper/2509.23050/) (2025)
- [Persistent Topological Features in Large Language Models](https://mlsanity.com/paper/2410.11042/) (2024)
- [Looking Beyond The Top-1: Transformers Determine Top Tokens In Order](https://mlsanity.com/paper/2410.20210/) (2024)
- [When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding](https://mlsanity.com/paper/2506.05551/) (2025)
- [Multimodal LLMs as Customized Reward Models for Text-to-Image Generation](https://mlsanity.com/paper/2507.21391/) (2025)
- [MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoning](https://mlsanity.com/paper/2508.20412/) (2025)
- [From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs](https://mlsanity.com/paper/2505.14368/) (2025)
- [AttenMIA: LLM Membership Inference Attack through Attention Signals](https://mlsanity.com/paper/2601.18110/) (2026)
- [Attention Basin: Why Contextual Position Matters in Large Language Models](https://mlsanity.com/paper/2508.05128/) (2025)
- [Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models](https://mlsanity.com/paper/2506.07424/) (2025)
- [SVAgent: AI Agent for Hardware Security Verification Assertion](https://mlsanity.com/paper/2507.16203/) (2025)
- [Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning](https://mlsanity.com/paper/2510.13832/) (2025)
- [PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision--Language Models](https://mlsanity.com/paper/2511.11502/) (2025)
- [Speech Recognition Model Improves Text-to-Speech Synthesis using Fine-Grained Reward](https://mlsanity.com/paper/2511.17555/) (2025)
- [Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility](https://mlsanity.com/paper/2605.06105/) (2026)
- [Hopscotch: Discovering and Skipping Redundancies in Language Models](https://mlsanity.com/paper/2506.03303/) (2025)
- [ShishuLM : Achieving Optimal and Efficient Parameterization with Low Attention Transformer Models](https://mlsanity.com/paper/2510.13860/) (2025)
- [Entropy and Attention Dynamics in Small Language Models: A Trace-Level Structural Analysis on the TruthfulQA Benchmark](https://mlsanity.com/paper/2604.03589/) (2026)
- [Deep Pre-Alignment for VLMs](https://mlsanity.com/paper/2605.15300/) (2026)
- [EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering](https://mlsanity.com/paper/2606.06906/) (2026)

## Abstract (arXiv)

In decoder-based LLMs, the representation of a given layer serves two purposes: as input to the next layer during the computation of the current token; and as input to the attention mechanism of future tokens. In this work, we show that the importance of the latter role might be overestimated. To show that, we start by manipulating the representations of previous tokens; e.g. by replacing the hidden states at some layer k with random vectors. Our experimenting with four LLMs and four tasks show that this operation often leads to small to negligible drop in performance. Importantly, this happens if the manipulation occurs in the top part of the model-k is in the final 30-50% of the layers. In contrast, doing the same manipulation in earlier layers might lead to chance level performance. We continue by switching the hidden state of certain tokens with hidden states of other tokens from another prompt; e.g., replacing the word "Italy" with "France" in "What is the capital of Italy?". We find that when applying this switch in the top 1/3 of the model, the model ignores it (answering "Rome"). However if we apply it before, the model conforms to the switch ("Paris"). Our results hint at a two stage process in transformer-based LLMs: the first part gathers input from previous tokens, while the second mainly processes that information internally.