mlsanityData to 7 Oct 2026

Exploring Context Window of Large Language Models via Decomposed Positional Vectors

arXiv 2405.18009 ↗First posted 28 May 2024cs.CL cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can effectively extend the context window length.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 29 of its 33 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

29 of 33 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
17 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,112
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2024Yi: Open Foundation Models by 01.AIcited by 706
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  6. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  7. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  8. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  9. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  10. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  11. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  12. 2024Length Generalization of Causal Transformers without Position Encodingcited by 46
  13. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
  14. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  15. 2024REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answeringcited by 28
  16. 2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23
  17. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 5.

18 papers.

Sort:

By area:

  1. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  2. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37
  3. 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 23
  4. 2025LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillationcited by 14
  5. 2025ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checkingcited by 9
  6. 2025Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applicationscited by 9
  7. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  8. 2025PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Samplingcited by 6
  9. 2025LogiDebrief: A Signal-Temporal Logic based Automated Debriefing Approach with Large Language Models Integrationcited by 5
  10. 2024Disentangling Latent Shifts of In-Context Learning with Weak Supervisioncited by 3
  11. 2025Evaluating Large Language Models in Vulnerability Detection Under Variable Context Windowscited by 2
  12. 2025Context-aware Biases for Length Extrapolationcited by 2
6 more
  1. 2025Real-time Spatial Retrieval Augmented Generation for Urban Environmentscited by 1
  2. 2026Empowering 9-1-1 Calltaking Training with Generative AI: Experiences and Lessons Learnedcited by 1
  3. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inferencecited by 1
  4. 2026GPT-5 vs Other LLMs in Long Short-Context Performancecited by 0
  5. 2026Can LLMs Reason Like Automated Theorem Provers for Rust Verification? VCoT-Bench: Evaluating via Verification Chain of Thoughtcited by 0
  6. 2026ZetaGPT: A Reference Implementation of Positional--Encoding--Free State--Space--Attention Language Modelscited by 0