SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
Not on the map. Why?
Abstract (arXiv)
While many advanced LLMs are designed to handle long sequence data, we can still observe notable quality degradation even within the sequence limit. In this work, we introduce a novel approach called Scaling to Emphasize Attention for Long-context retrieval (SEAL), which enhances the retrieval performance of large language models (LLMs) over long contexts. We observe that specific attention heads are closely tied to long-context retrieval, showing positive or negative correlation with retrieval scores, and adjusting the strength of these heads boosts the quality of LLMs in long context by a large margin. Built on this insight, we propose a learning-based mechanism that leverages generated data to emphasize these heads. By applying SEAL, we achieve significant improvements in long-context retrieval performance across various tasks and models. Additionally, when combined with existing training-free context extension techniques, SEAL extends the contextual limits of LLMs while maintaining highly reliable outputs.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 25 Jan 2025.
Builds on
31 of 43 references.
References · 31
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023Mistral 7Bcited by 3,164
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
19 more
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2024Mixtral of Expertscited by 1,823
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
- 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
- 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
- 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
- 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
- 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
- 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
- 2023Retrieval meets Long Context Large Language Modelscited by 115
- 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
- 2023Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Trainingcited by 32
- 2023Outlier Dimensions Encode Task-Specific Knowledgecited by 15
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.