mlsanityData to 7 Oct 2026

A Length-Extrapolatable Transformer

arXiv 2212.10554 ↗First posted 20 Dec 2022cs.CLcited by 123

Not on the map. Why?

Abstract (arXiv)

Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define attention resolution as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution. We evaluate different Transformer variants with language modeling. Experimental results show that our model achieves strong performance in both interpolation and extrapolation settings. The code will be available at https://aka.ms/LeX-Transformer.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

31 of 42 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
19 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2020Rethinking Attention with Performerscited by 1,584
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2018Self-Attention with Relative Position Representationscited by 1,112
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  11. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  12. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  13. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  14. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  15. 2022Block-Recurrent Transformerscited by 107
  16. 2021Position Information in Transformers: An Overviewcited by 103
  17. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  18. 2021Simple Local Attentions Remain Competitive for Long-Context Taskscited by 25
  19. 2022TorchScale: Transformers at Scalecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 31; 2024: 48; 2025: 29; 2026: 15.

123 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2024Chronos: Learning the Language of Time Seriescited by 712
  5. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  6. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  7. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  8. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  9. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  10. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  11. 2023Efficient Large Language Models: A Surveycited by 183
  12. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
38 more
  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  2. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  3. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  4. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  5. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  6. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 89
  7. 2024Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scalecited by 87
  8. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  9. 2024FiT: Flexible Vision Transformer for Diffusion Modelcited by 84
  10. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  11. 2023GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformerscited by 75
  12. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  13. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  14. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  15. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  16. 2025This Time is Different: An Observability Perspective on Time Series Foundation Modelscited by 57
  17. 2023Large Language Models for Compiler Optimizationcited by 52
  18. 2024Base of RoPE Bounds Context Lengthcited by 49
  19. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  20. 2024Toto: Time Series Optimized Transformer for Observabilitycited by 44
  21. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  22. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
  23. 2024FengWu-GHR: Learning the Kilometer-scale Medium-range Global Weather Forecastingcited by 30
  24. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  25. 2023DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Trainingcited by 28
  26. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  27. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  28. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  29. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  30. 2023BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Modelcited by 19
  31. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  32. 2023Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networkscited by 18
  33. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  34. 2024Transforming the Bootstrap: Using Transformers to Compute Scattering Amplitudes in Planar N = 4 Super Yang-Mills Theorycited by 17
  35. 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
  36. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  37. 2024Bass Accompaniment Generation via Latent Diffusioncited by 16
  38. 2023M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Modelscited by 15

50 most cited of 123 All 123 (JSON)