mlsanityData to 14 Oct 2026

Extending Llama-3's Context Ten-Fold Overnight

arXiv 2404.19553 ↗First posted 30 Apr 2024cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G) GPU machine. The resulted model exhibits superior performances across a broad range of evaluation tasks, such as NIHS, topic retrieval, and long-context language understanding; meanwhile, it also well preserves the original capability over short contexts. The dramatic context extension is mainly attributed to merely 3.5K synthetic training samples generated by GPT-4 , which indicates the LLMs' inherent (yet largely underestimated) potential to extend its original context length. In fact, the context length could be extended far beyond 80K with more computation resources. Therefore, the team will publicly release the entire resources (including data, model, data generation pipeline, training code) so as to facilitate the future research from the community: urlhttps://github.com/FlagOpen/FlagEmbedding.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 6 of its 9 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

6 of 9 references.

References · 6

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  3. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  4. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  5. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  6. 2024Long Context Compression with Activation Beaconcited by 104

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 5; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 119
  2. 2024How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensionscited by 52
  3. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  4. 2024Are Long-LLMs A Necessity For Long-Context Tasks?cited by 22
  5. 2024Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Modelscited by 20
  6. 2025Gradient-Based Model Fingerprinting for LLM Similarity Detection and Family Classificationcited by 14
  7. 2025SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimizationcited by 10
  8. 2024LOGO -- Long cOntext aliGnment via efficient preference Optimizationcited by 9
  9. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retentioncited by 7
  10. 2025Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoningcited by 6
  11. 2024Attention Overflow: Language Model Input Blur during Long-Context Missing Items Recommendationcited by 2
  12. 2025Flora: Effortless Context Construction to Arbitrary Length and Scalecited by 2
3 more
  1. 2024Two are better than one: Context window extension with multi-grained self-injectioncited by 1
  2. 2024Boosting Long-Context Management via Query-Guided Activation Refillingcited by 1
  3. 2026Stacked from One: Multi-Scale Self-Injection for Context Window Extensioncited by 0