mlsanityData to 7 Oct 2026

LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models

arXiv 2311.04879 ↗First posted 8 Nov 2023cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

We present LongQLoRA, an efficient and effective method to extend context length of large language models with less training resources. LongQLoRA combines the advantages of Position Interpolation, QLoRA and Shift Short Attention of LongLoRA. With a single 32GB V100 GPU, LongQLoRA can extend the context length of LLaMA2 7B and 13B from 4096 to 8192 and even to 12k within 1000 finetuning steps. LongQLoRA achieves competitive perplexity performance on PG19 and Proof-pile datasets, our model outperforms LongLoRA and is very close to MPT-7B-8K within the evaluation context length of 8192. We collect and build 39k long instruction data to extend context length of Vicuna-13B from 4096 to 8192 and achieve good performance both in long and short context generation task. We also do some ablation experiments to study the effect of LoRA rank, finetuning steps and attention patterns in inference.The model weights, training data and code are avaliable at https://github.com/yangjianxin1/LongQLoRA.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 14 of its 21 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

14 of 21 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2016Layer Normalizationcited by 5,579
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
2 more
  1. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  2. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 2.

12 papers.

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  4. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  5. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  6. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  7. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  8. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  9. 2024Schrodinger's Memory: Large Language Modelscited by 2
  10. 2024FltLM: An Intergrated Long-Context Large Language Model for Effective Context Filtering and Understandingcited by 2
  11. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Modelscited by 1
  12. 2026ICICLE: Expanding Retrieval with In-Context Documentscited by 0