mlsanityData to 14 Oct 2026

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

arXiv 2309.10400 ↗First posted 19 Sep 2023cs.CL cs.LGcited by 111

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are trained with a pre-defined context length, restricting their use in scenarios requiring long inputs. Previous efforts for adapting LLMs to a longer length usually requires fine-tuning with this target length (Full-length fine-tuning), suffering intensive training cost. To decouple train length from target length for efficient context window extension, we propose Positional Skip-wisE (PoSE) training that smartly simulates long inputs using a fixed context window. This is achieved by first dividing the original context window into several chunks, then designing distinct skipping bias terms to manipulate the position indices of each chunk. These bias terms and the lengths of each chunk are altered for every training example, allowing the model to adapt to all positions within target length. Experimental results show that PoSE greatly reduces memory and time overhead compared with Full-length fine-tuning, with minimal impact on performance. Leveraging this advantage, we have successfully extended the LLaMA model to 128k tokens using a 2k training context window. Furthermore, we empirically confirm that PoSE is compatible with all RoPE-based LLMs and position interpolation strategies. Notably, our method can potentially support infinite length, limited only by memory usage in inference. With ongoing progress for efficient inference, we believe PoSE can further scale the context window beyond 128k.

Key facts. Cited by 111 papers in the map (data through 14 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

29 of 42 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
17 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  5. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  6. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  7. 2021Efficient Attentions for Long Document Summarizationcited by 345
  8. 2022Memorizing Transformerscited by 226
  9. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  10. 2022Recurrent Memory Transformercited by 203
  11. 2023Augmenting Language Models with Long-Term Memorycited by 189
  12. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  13. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  14. 2022A Length-Extrapolatable Transformercited by 127
  15. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  16. 2023In-Context Learning with Many Demonstration Examplescited by 39
  17. 2022Fine-Grained Distillation for Long Document Retrievalcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 51; 2025: 31; 2026: 20.

111 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  4. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  5. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
  6. 2023Improving Text Embeddings with Large Language Modelscited by 303
  7. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  8. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  9. 2023Efficient Large Language Models: A Surveycited by 186
  10. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  11. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  12. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
38 more
  1. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  2. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  3. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 108
  4. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  5. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  6. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  7. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  8. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  9. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  10. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  11. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 48
  12. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 43
  13. 2024With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generationcited by 37
  14. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 33
  15. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  16. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 31
  17. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 29
  18. 2024LongGenBench: Long-context Generation Benchmarkcited by 28
  19. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  20. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
  21. 2024Resonance RoPE: Improving Context Length Generalization of Large Language Modelscited by 27
  22. 2024Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculumcited by 26
  23. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 24
  24. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  25. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 23
  26. 2024Long Context Alignment with Short Instructions and Synthesized Positionscited by 22
  27. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervisioncited by 21
  28. 2024LoCoCo: Dropping In Convolutions for Long Context Compressioncited by 19
  29. 2023Large Search Model: Redefining Search Stack in the Era of LLMscited by 18
  30. 2023Empowering Working Memory for Large Language Model Agentscited by 18
  31. 2024Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Modelcited by 16
  32. 2024An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encodingcited by 16
  33. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Datacited by 16
  34. 2025Native-Resolution Image Synthesiscited by 15
  35. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  36. 2025LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillationcited by 14
  37. 2024Towards LifeSpan Cognitive Systemscited by 13
  38. 2024ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coveragecited by 12

50 most cited of 111 All 111 (JSON)