# From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data

arXiv: https://arxiv.org/abs/2406.19292 · HTML: https://mlsanity.com/paper/2406.19292/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 34 of its 40 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 30 of 34)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024)
- [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://mlsanity.com/paper/2307.11088/) (2023)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [Data Engineering for Scaling Language Models to 128K Context](https://mlsanity.com/paper/2402.10171/) (2024)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://mlsanity.com/paper/2402.14848/) (2024)
- [LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning](https://mlsanity.com/paper/2401.01325/) (2024)
- [Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading](https://mlsanity.com/paper/2310.05029/) (2023)
- [Teaching Arithmetic to Small Transformers](https://mlsanity.com/paper/2307.03381/) (2023)
- [Make Your LLM Fully Utilize the Context](https://mlsanity.com/paper/2404.16811/) (2024)
- [Retrieval meets Long Context Large Language Models](https://mlsanity.com/paper/2310.03025/) (2023)
- [Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance](https://mlsanity.com/paper/2305.17306/) (2023)
- [DataPerf: Benchmarks for Data-Centric AI Development](https://mlsanity.com/paper/2207.10062/) (2022)
- [Attention Sorting Combats Recency Bias In Long Context Language Models](https://mlsanity.com/paper/2310.01427/) (2023)
- [Transformers Can Achieve Length Generalization But Not Robustly](https://mlsanity.com/paper/2402.09371/) (2024)

## Built on by (most cited first, 18 of 18)

- [How to Train Long-Context Language Models (Effectively)](https://mlsanity.com/paper/2410.02660/) (2024)
- [Context Length Alone Hurts LLM Performance Despite Perfect Retrieval](https://mlsanity.com/paper/2510.05381/) (2025)
- [RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code](https://mlsanity.com/paper/2503.07832/) (2025)
- [Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs](https://mlsanity.com/paper/2509.21361/) (2025)
- [LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs](https://mlsanity.com/paper/2503.02502/) (2025)
- [Understanding Synthetic Context Extension via Retrieval Heads](https://mlsanity.com/paper/2410.22316/) (2024)
- [CLIPPER: Compression enables long-context synthetic data generation](https://mlsanity.com/paper/2502.14854/) (2025)
- [Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models](https://mlsanity.com/paper/2409.04774/) (2024)
- [WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale](https://mlsanity.com/paper/2502.16684/) (2025)
- [Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems](https://mlsanity.com/paper/2411.01101/) (2024)
- [Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find](https://mlsanity.com/paper/2505.18148/) (2025)
- [Decoding Hate: Exploring Language Models' Reactions to Hate Speech](https://mlsanity.com/paper/2410.00775/) (2024)
- [Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context](https://mlsanity.com/paper/2410.07103/) (2024)
- [Single-Pass Document Scanning for Question Answering](https://mlsanity.com/paper/2504.03101/) (2025)
- [Learning from Synthetic Data Improves Multi-hop Reasoning](https://mlsanity.com/paper/2603.02091/) (2026)
- [PERK: Long-Context Reasoning as Test-Time Learning](https://mlsanity.com/paper/2507.06415/) (2025)
- [When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models](https://mlsanity.com/paper/2509.18762/) (2025)
- [In-Context Binding Capacity in Language Models](https://mlsanity.com/paper/2609.30634/) (2026)

## Abstract (arXiv)

Recent studies have shown that Large Language Models (LLMs) struggle to accurately retrieve information and maintain reasoning capabilities when processing long-context inputs. To address these limitations, we propose a finetuning approach utilizing a carefully designed synthetic dataset comprising numerical key-value retrieval tasks. Our experiments on models like GPT-3.5 Turbo and Mistral 7B demonstrate that finetuning LLMs on this dataset significantly improves LLMs' information retrieval and reasoning capabilities in longer-context settings. We present an analysis of the finetuned models, illustrating the transfer of skills from synthetic to real task evaluations (e.g., 10.5% improvement on 20 documents MDQA at position 10 for GPT-3.5 Turbo). We also find that finetuned LLMs' performance on general benchmarks remains almost constant while LLMs finetuned on other baseline long-context augmentation data can encourage hallucination (e.g., on TriviaQA, Mistral 7B finetuned on our synthetic data cause no performance drop while other baseline data can cause a drop that ranges from 2.33% to 6.19%). Our study highlights the potential of finetuning on synthetic data for improving the performance of LLMs on longer-context tasks.