mlsanityData to 7 Oct 2026

Data Management For Training Large Language Models: A Survey

arXiv 2312.01700 ↗First posted 4 Dec 2023cs.CL cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Data plays a fundamental role in training Large Language Models (LLMs). Efficient data management, particularly in formulating a well-suited training dataset, is significant for enhancing model performance and improving training efficiency during pretraining and supervised fine-tuning stages. Despite the considerable importance of data management, the underlying mechanism of current prominent practices are still unknown. Consequently, the exploration of data management has attracted more and more attention among the research community. This survey aims to provide a comprehensive overview of current research in data management within both the pretraining and supervised fine-tuning stages of LLMs, covering various aspects of data management strategy design. Looking into the future, we extrapolate existing challenges and outline promising directions for development in this field. Therefore, this survey serves as a guiding resource for practitioners aspiring to construct powerful LLMs through efficient data management practices. The collection of the latest papers is available at https://github.com/ZigeW/data_management_LLM.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 98 of its 154 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

98 of 154 references.

References · 98

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
38 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2023LIMA: Less Is More for Alignmentcited by 1,080
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  13. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  14. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  15. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  16. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  17. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  18. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  19. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  20. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  21. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  22. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  23. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  24. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  25. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  26. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  27. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  28. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  29. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  30. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  31. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  32. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  33. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  34. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  35. 2019Measuring Bias in Contextualized Word Representationscited by 275
  36. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  37. 2023The False Promise of Imitating Proprietary LLMscited by 254
  38. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 4; 2026: 2.

17 papers.

Sort:

By area:

  1. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  2. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  3. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 42
  4. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 23
  5. 2024Balanced Data Sampling for Language Model Training with Clusteringcited by 22
  6. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  7. 2025What Makes Diffusion Language Models Super Data Learners?cited by 8
  8. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  9. 2024Language Models as Continuous Self-Evolving Data Engineerscited by 6
  10. 2024Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularitycited by 4
  11. 2025A Survey of LLM × DATAcited by 4
  12. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
5 more
  1. 2024Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Modelscited by 1
  2. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  3. 20252K-Characters-10K-Stories: A Quality-Gated Stylized Narrative Dataset with Disentangled Control and Sequence Consistencycited by 0
  4. 2025Toward Agentic Environments: GenAI and the Convergence of AI, Sustainability, and Human-Centric Spacescited by 0
  5. 2026A Unified Framework for In-Context Learning with Causal and Masked Language Modelscited by 0