mlsanityData to 7 Oct 2026

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

arXiv 2309.12307 ↗First posted 21 Sep 2023cs.CL cs.AI cs.LGcited by 243

Not on the map. Why?

Cited by 29 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present LongLoRA, an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs), with limited computation cost. Typically, training LLMs with long context sizes is computationally expensive, requiring extensive training hours and GPU resources. For example, training on the context length of 8192 needs 16x computational costs in self-attention layers as that of 2048. In this paper, we speed up the context extension of LLMs in two aspects. On the one hand, although dense global attention is needed during inference, fine-tuning the model can be effectively and efficiently done by sparse local attention. The proposed shifted sparse attention effectively enables context extension, leading to non-trivial computation saving with similar performance to fine-tuning with vanilla attention. Particularly, it can be implemented with only two lines of code in training, while being optional in inference. On the other hand, we revisit the parameter-efficient fine-tuning regime for context expansion. Notably, we find that LoRA for context extension works well under the premise of trainable embedding and normalization. LongLoRA combines this improved LoRA with S^2-Attn. LongLoRA demonstrates strong empirical results on various tasks on Llama2 models from 7B/13B to 70B. LongLoRA extends Llama2 7B from 4k context to 100k, or Llama2 70B to 32k on a single 8x A100 machine. LongLoRA extends models' context while retaining their original architectures, and is compatible with most existing techniques, like Flash-Attention2. In addition, we further conduct supervised fine-tuning with LongLoRA and our long instruction-following LongAlpaca dataset.

Key facts. Cited by 243 papers in the map (data through 7 Oct 2026). Builds on 39 of its 61 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

39 of 61 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2016Layer Normalizationcited by 5,579
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
27 more
  1. 2020Longformer: The Long-Document Transformercited by 3,362
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2020Reformer: The Efficient Transformercited by 1,658
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  13. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  14. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  15. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  16. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  17. 2022Memorizing Transformerscited by 219
  18. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  19. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  20. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  21. 2022Recurrent Memory Transformercited by 197
  22. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  23. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  24. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  25. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  26. 2020GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimationcited by 36
  27. 2022Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Modelscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 112; 2025: 72; 2026: 41.

243 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  5. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  6. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  7. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  8. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  9. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  10. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  11. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  12. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
38 more
  1. 2025LongLive: Real-time Interactive Long Video Generationcited by 237
  2. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  3. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  4. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  5. 2023Efficient Large Language Models: A Surveycited by 183
  6. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  7. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  8. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  9. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  10. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  11. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  12. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  13. 2024Long Context Compression with Activation Beaconcited by 101
  14. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  15. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  16. 2025Scaling RL to Long Videoscited by 83
  17. 2024From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMscited by 77
  18. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  19. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  20. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  21. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  22. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  23. 2024A Survey on LoRA of Large Language Modelscited by 67
  24. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  25. 2025SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Trainingcited by 63
  26. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  27. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Datacited by 61
  28. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  29. 2024Depth Any Video with Scalable Synthetic Datacited by 57
  30. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  31. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  32. 2024ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilitiescited by 52
  33. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  34. 2023End-to-End Temporal Action Detection with 1B Parameters Across 1000 Framescited by 47
  35. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 47
  36. 2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 46
  37. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  38. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45

50 most cited of 243 All 243 (JSON)