mlsanityData to 14 Oct 2026

Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model

arXiv 2405.19846 ↗First posted 30 May 2024cs.CL cs.AIcited by 16

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Recent advancements in large language models (LLMs) have highlighted the importance of extending context lengths for handling complex tasks. While traditional methods for training on long contexts often use filtered long documents, these approaches lead to domain imbalances, limiting model performance. To address this, techniques like random document concatenation (Standard) and similarity-based methods (KNN, ICLM) have been developed. However, they either sacrifice semantic coherence or diversity. To balance both aspects, we introduce Quest, a query-centric data synthesis method aggregating semantically relevant yet diverse documents. Quest uses a generative model to predict potential queries for each document, grouping documents with similar queries and keywords. Extensive experiments demonstrate Quest's superior performance on long-context tasks, achieving remarkable results with context lengths of up to 1M tokens and confirming its scalability across various model sizes.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 54 of its 76 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

54 of 76 references.

References · 54

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
38 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  13. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  14. 2024Yi: Open Foundation Models by 01.AIcited by 719
  15. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  16. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  17. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  18. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  19. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  20. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  21. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  22. 2019Document Expansion by Query Predictioncited by 231
  23. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  24. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  25. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  26. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  27. 2022Revisiting Neural Scaling Laws in Language and Visioncited by 132
  28. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  29. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  30. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  31. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  32. 2021Learning Passage Impacts for Inverted Indexescited by 68
  33. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 44
  34. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33
  35. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28
  36. 2024Unraveling the Mystery of Scaling Laws: Part Icited by 24
  37. 2023Resolving the Imbalance Issue in Hierarchical Disciplinary Topic Inference via LLM-based Data Augmentationcited by 11
  38. 2024Temporal Scaling Law for Large Language Modelscited by 11

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 4.

16 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  2. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  3. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  4. 2024What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practicescited by 14
  5. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  6. 2026Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learningcited by 8
  7. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  8. 2025LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructionscited by 6
  9. 2025EntropyLong: Effective Long-Context Training via Predictive Uncertaintycited by 5
  10. 2025Revisiting Long-context Modeling from Context Denoising Perspectivecited by 5
  11. 2024Bootstrap Your Own Context Lengthcited by 4
  12. 2025LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMscited by 3
4 more
  1. 2026Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shapingcited by 3
  2. 2025Libra: Large Chinese-based Safeguard for AI Contentcited by 2
  3. 2026PolicyLong: Towards On-Policy Context Extensioncited by 0
  4. 2026TrajLong: Co-Designing Agentic and Long-Context Supervision for Mid-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  2. 2024A Survey of Query Optimization in Large Language Models
  3. 2023MILL: Mutual Verification with Large Language Models for Zero-Shot Query Expansion
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
  5. 2025MacRAG: Compress, Slice, and Scale-up for Multi-Scale Adaptive Context RAG
  6. 2024Enhancing Cloud-Based Large Language Model Processing with Elasticsearch and Transformer Models
  7. 2024On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models
  8. 2025Less LLM, More Documents: Searching for Improved RAG