# Extending Llama-3's Context Ten-Fold Overnight

arXiv: https://arxiv.org/abs/2404.19553 · HTML: https://mlsanity.com/paper/2404.19553/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 9 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Builds on (most cited first, 6 of 6)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [Data Engineering for Scaling Language Models to 128K Context](https://mlsanity.com/paper/2402.10171/) (2024)
- [Long Context Compression with Activation Beacon](https://mlsanity.com/paper/2401.03462/) (2024)

## Built on by (most cited first, 15 of 15)

- [MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation](https://mlsanity.com/paper/2409.05591/) (2024)
- [How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions](https://mlsanity.com/paper/2406.14805/) (2024)
- [Large Language Models Can Self-Improve in Long-context Reasoning](https://mlsanity.com/paper/2411.08147/) (2024)
- [Are Long-LLMs A Necessity For Long-Context Tasks?](https://mlsanity.com/paper/2405.15318/) (2024)
- [Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models](https://mlsanity.com/paper/2409.20222/) (2024)
- [Gradient-Based Model Fingerprinting for LLM Similarity Detection and Family Classification](https://mlsanity.com/paper/2506.01631/) (2025)
- [SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization](https://mlsanity.com/paper/2505.11166/) (2025)
- [LOGO -- Long cOntext aliGnment via efficient preference Optimization](https://mlsanity.com/paper/2410.18533/) (2024)
- [Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention](https://mlsanity.com/paper/2505.15774/) (2025)
- [Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning](https://mlsanity.com/paper/2502.13127/) (2025)
- [Attention Overflow: Language Model Input Blur during Long-Context Missing Items Recommendation](https://mlsanity.com/paper/2407.13481/) (2024)
- [Flora: Effortless Context Construction to Arbitrary Length and Scale](https://mlsanity.com/paper/2507.19786/) (2025)
- [Two are better than one: Context window extension with multi-grained self-injection](https://mlsanity.com/paper/2410.19318/) (2024)
- [Boosting Long-Context Management via Query-Guided Activation Refilling](https://mlsanity.com/paper/2412.12486/) (2024)
- [Stacked from One: Multi-Scale Self-Injection for Context Window Extension](https://mlsanity.com/paper/2603.04759/) (2026)

## Abstract (arXiv)

We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G) GPU machine. The resulted model exhibits superior performances across a broad range of evaluation tasks, such as NIHS, topic retrieval, and long-context language understanding; meanwhile, it also well preserves the original capability over short contexts. The dramatic context extension is mainly attributed to merely 3.5K synthetic training samples generated by GPT-4 , which indicates the LLMs' inherent (yet largely underestimated) potential to extend its original context length. In fact, the context length could be extended far beyond 80K with more computation resources. Therefore, the team will publicly release the entire resources (including data, model, data generation pipeline, training code) so as to facilitate the future research from the community: urlhttps://github.com/FlagOpen/FlagEmbedding.