Pay Attention to What You Need
Not on the map. Why?
Abstract (arXiv)
Although large language models (LLMs) have achieved significant success in natural language processing, they still struggle with long-context comprehension. Traditional approaches to mitigating this issue typically rely on fine-tuning or retraining, which is both resource-intensive and challenging to deploy in lightweight industrial settings. In this paper, we investigate the potential to accomplish this without any additional resources. Through an in-depth study of the attention mechanism in LLMs, we propose a method called Scaled ReAttention (SRA) to strengthen LLMs' ability to interpret and retrieve information by strategically manipulating their attention scores during inference. Through extensive experiments, we demonstrate that integrating SRA significantly boosts LLMs' performance on a variety of downstream tasks, highlighting its practical potential for enhancing language understanding without incurring the overhead of traditional training.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 75 references that are in the map. First posted on arXiv 25 Jul 2023.
Builds on
30 of 75 references.
References · 30
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
18 more
- 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
- 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
- 2017The NarrativeQA Reading Comprehension Challengecited by 590
- 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
- 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
- 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
- 2021Efficient Attentions for Long Document Summarizationcited by 342
- 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
- 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
- 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
- 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
- 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83
- 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
- 2023VCSUM: A Versatile Chinese Meeting Summarization Datasetcited by 18
- 2020Deep Reinforcement Learning of Transition Statescited by 8
Built on by
Citing papers per year: 2023: 1.
1 paper.