mlsanityData to 14 Oct 2026

LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs

arXiv 2509.15568 ↗First posted 19 Sep 2025cs.CL cs.AIcited by 3

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

High-quality long-context data is essential for training large language models (LLMs) capable of processing extensive documents, yet existing synthesis approaches using relevance-based aggregation face challenges of computational efficiency. We present LiteLong, a resource-efficient method for synthesizing long-context data through structured topic organization and multi-agent debate. Our approach leverages the BISAC book classification system to provide a comprehensive hierarchical topic organization, and then employs a debate mechanism with multiple LLMs to generate diverse, high-quality topics within this structure. For each topic, we use lightweight BM25 retrieval to obtain relevant documents and concatenate them into 128K-token training samples. Experiments on HELMET and Ruler benchmarks demonstrate that LiteLong achieves competitive long-context performance and can seamlessly integrate with other long-dependency enhancement methods. LiteLong makes high-quality long-context data synthesis more accessible by reducing both computational and data engineering costs, facilitating further research in long-context language training.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 29 of its 35 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

29 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model30 May

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  11. 2025Gemma 3 Technical Reportcited by 1,824
  12. 2024Mixtral of Expertscited by 1,823
16 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2024Yi: Open Foundation Models by 01.AIcited by 719
  6. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  7. 2018AI safety via debatecited by 400
  8. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  9. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
  10. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  11. 2018Question Answering by Reasoning Across Documents with Graph Convolutional Networkscited by 56
  12. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33
  13. 2023Less is More for Long Document Summary Evaluation by LLMscited by 28
  14. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 20
  15. 2024Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Modelscited by 9
  16. 2025dots.llm1 Technical Reportcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Trainingcited by 3
  2. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 2
  3. 2026PolicyLong: Towards On-Policy Context Extensioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
  2. 2025SagaScale: A Realistic, Scalable, and High-Quality Long-Context Benchmark Built from Full-Length Novels
  3. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
  4. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
  5. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Tasks
  6. 2025LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
  7. 2025Systematic Evaluation of Optimization Techniques for Long-Context Language Models
  8. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaboration