mlsanityData to 7 Oct 2026

From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data

arXiv 2406.19292 ↗First posted 27 Jun 2024cs.LG cs.AI cs.CLcited by 18

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Recent studies have shown that Large Language Models (LLMs) struggle to accurately retrieve information and maintain reasoning capabilities when processing long-context inputs. To address these limitations, we propose a finetuning approach utilizing a carefully designed synthetic dataset comprising numerical key-value retrieval tasks. Our experiments on models like GPT-3.5 Turbo and Mistral 7B demonstrate that finetuning LLMs on this dataset significantly improves LLMs' information retrieval and reasoning capabilities in longer-context settings. We present an analysis of the finetuned models, illustrating the transfer of skills from synthetic to real task evaluations (e.g., 10.5% improvement on 20 documents MDQA at position 10 for GPT-3.5 Turbo). We also find that finetuned LLMs' performance on general benchmarks remains almost constant while LLMs finetuned on other baseline long-context augmentation data can encourage hallucination (e.g., on TriviaQA, Mistral 7B finetuned on our synthetic data cause no performance drop while other baseline data can cause a drop that ranges from 2.33% to 6.19%). Our study highlights the potential of finetuning on synthetic data for improving the performance of LLMs on longer-context tasks.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 34 of its 40 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

34 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models19 Feb
  2. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance26 May

Other references · 32

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023Mistral 7Bcited by 3,065
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
20 more
  1. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  2. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  3. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  4. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  5. 2024A Survey on Data Selection for Language Modelscited by 230
  6. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  7. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  8. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  9. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  10. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  11. 2023Teaching Arithmetic to Small Transformerscited by 121
  12. 2024Make Your LLM Fully Utilize the Contextcited by 118
  13. 2023Retrieval meets Long Context Large Language Modelscited by 114
  14. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  15. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  16. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  17. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  18. 2023Data-centric AI: Perspectives and Challengescited by 43
  19. 2023Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Modelscited by 30
  20. 2023Training With "Paraphrasing the Original Text" Teaches LLM to Better Retrieve in Long-context Taskscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 10; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  2. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 118
  3. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  4. 2025LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMscited by 11
  5. 2025Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMscited by 11
  6. 2024Understanding Synthetic Context Extension via Retrieval Headscited by 10
  7. 2024Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Modelscited by 9
  8. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 9
  9. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  10. 2024Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problemscited by 7
  11. 2025Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Findcited by 7
  12. 2024Decoding Hate: Exploring Language Models' Reactions to Hate Speechcited by 6
6 more
  1. 2024Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Contextcited by 6
  2. 2025Single-Pass Document Scanning for Question Answeringcited by 6
  3. 2026Learning from Synthetic Data Improves Multi-hop Reasoningcited by 6
  4. 2025PERK: Long-Context Reasoning as Test-Time Learningcited by 4
  5. 2025When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Modelscited by 1
  6. 2026In-Context Binding Capacity in Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities
  2. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  3. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
  4. 2026Adaptive Retrieval helps Reasoning in LLMs -- but mostly if it's not used
  5. 2025Finetune-RAG: Fine-Tuning Language Models to Resist Hallucination in Retrieval-Augmented Generation
  6. 2025Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
  7. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge
  8. 2024Efficacy of Synthetic Data as a Benchmark