mlsanityData to 14 Oct 2026

Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey

arXiv 2311.12351 ↗First posted 21 Nov 2023cs.CL cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

Transformer-based Large Language Models (LLMs) have been applied in diverse areas such as knowledge bases, human interfaces, and dynamic agents, and marking a stride towards achieving Artificial General Intelligence (AGI). However, current LLMs are predominantly pretrained on short text snippets, which compromises their effectiveness in processing the long-context prompts that are frequently encountered in practical scenarios. This article offers a comprehensive survey of the recent advancement in Transformer-based LLM architectures aimed at enhancing the long-context capabilities of LLMs throughout the entire model lifecycle, from pre-training through to inference. We first delineate and analyze the problems of handling long-context input and output with the current Transformer-based models. We then provide a taxonomy and the landscape of upgrades on Transformer architecture to solve these problems. Afterwards, we provide an investigation on wildly used evaluation necessities tailored for long-context LLMs, including datasets, metrics, and baseline models, as well as optimization toolkits such as libraries, frameworks, and compilers to boost the efficacy of LLMs across different stages in runtime. Finally, we discuss the challenges and potential avenues for future research. A curated repository of relevant literature, continuously updated, is available at https://github.com/Strivin0311/long-llms-learning.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Builds on 178 of its 255 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

178 of 255 references.

References · 178

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  11. 2017Graph Attention Networkscited by 8,923
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
38 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2016Layer Normalizationcited by 5,669
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  13. 2020Longformer: The Long-Document Transformercited by 3,413
  14. 2023Mistral 7Bcited by 3,164
  15. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  16. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  17. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  18. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  19. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  20. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  21. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  22. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  23. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  24. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  25. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  26. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  27. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  28. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  29. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  30. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  31. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  32. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  33. 2020Reformer: The Efficient Transformercited by 1,668
  34. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  35. 2020Rethinking Attention with Performerscited by 1,601
  36. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  37. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  38. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463

50 most cited of 178 All 178 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 31; 2025: 27; 2026: 11.

72 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  2. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  3. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  4. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  5. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  6. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  7. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  8. 2024Length Generalization of Causal Transformers without Position Encodingcited by 47
  9. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 47
  10. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  11. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
  12. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
38 more
  1. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 32
  2. 2024LongGenBench: Long-context Generation Benchmarkcited by 28
  3. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  4. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
  5. 2024AppBench: Planning of Multiple APIs from Various APPs for Complex User Instructioncited by 24
  6. 2025Mid-Training of Large Language Models: A Surveycited by 21
  7. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 21
  8. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  9. 2023Learning Long Sequences in Spiking Neural Networkscited by 19
  10. 2024Counting-Stars: A Multi-evidence, Position-aware, and Scalable Benchmark for Evaluating Long-Context Large Language Modelscited by 17
  11. 2024Compressing Long Context for Enhancing RAG with AMR-based Concept Distillationcited by 16
  12. 2024SirLLM: Streaming Infinite Retentive LLMcited by 16
  13. 2024Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generationcited by 15
  14. 2024MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokenscited by 13
  15. 2023Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learningcited by 12
  16. 2024Efficient Streaming LLM for Speech Recognitioncited by 12
  17. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  18. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  19. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  20. 2024Can Large Language Models Detect Rumors on Social Media?cited by 9
  21. 2024StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Lossescited by 9
  22. 2024AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systemscited by 9
  23. 2025A Survey on Enhancing Causal Reasoning Ability of Large Language Modelscited by 8
  24. 2025NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tablescited by 8
  25. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  26. 2025SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokenscited by 8
  27. 2026VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memorycited by 6
  28. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Surveycited by 5
  29. 2025CoRank: LLM-Based Compact Reranking with Document Features for Scientific Retrievalcited by 5
  30. 2025FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUscited by 5
  31. 2024QuickLLaMA: Query-aware Inference Acceleration for Large Language Modelscited by 4
  32. 2024An Evolved Universal Transformer Memorycited by 4
  33. 2025The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architecturescited by 4
  34. 2025MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMscited by 4
  35. 2026MrRoPE: Mixed-radix Rotary Position Embeddingcited by 4
  36. 2024Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processingcited by 3
  37. 2025Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolationcited by 3
  38. 2025Weak-to-Strong GraphRAG: Aligning Weak Retrievers with Large Language Models for Graph-based Retrieval Augmented Generationcited by 3

50 most cited of 72 All 72 (JSON)