mlsanityData to 14 Oct 2026

A Survey on Efficient Large Language Model Training: From Data-centric Perspectives

arXiv 2510.25817 ↗First posted 29 Oct 2025cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Post-training of Large Language Models (LLMs) is crucial for unlocking their task generalization potential and domain-specific capabilities. However, the current LLM post-training paradigm faces significant data challenges, including the high costs of manual annotation and diminishing marginal returns on data scales. Therefore, achieving data-efficient post-training has become a key research question. In this paper, we present the first systematic survey of data-efficient LLM post-training from a data-centric perspective. We propose a taxonomy of data-efficient LLM post-training methods, covering data selection, data quality enhancement, synthetic data generation, data distillation and compression, and self-evolving data ecosystems. We summarize representative approaches in each category and outline future research directions. By examining the challenges in data-efficient LLM post-training, we highlight open problems and propose potential research avenues. We hope our work inspires further exploration into maximizing the potential of data utilization in large-scale model training. Paper List: https://github.com/luo-junyu/Awesome-Data-Efficient-LLM

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 70 of its 124 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

70 of 124 references.

References · 70

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2023Let's Verify Step by Stepcited by 3,789
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  7. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  8. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  9. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  10. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  11. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  12. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
38 more
  1. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  3. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  4. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  5. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  6. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  7. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  8. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  9. 2024Data-efficient Fine-tuning for LLM-based Recommendationcited by 104
  10. 2024ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoningcited by 103
  11. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 94
  12. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  13. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 79
  14. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 77
  15. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluationcited by 75
  16. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 70
  17. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 63
  18. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 55
  19. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  20. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  21. 2024Data is all you need: Finetuning LLMs for Chip Design via an Automated design-data augmentation frameworkcited by 36
  22. 2022Toxicity Detection with Generative Prompt-based Inferencecited by 35
  23. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 34
  24. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  25. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  26. 2024Improving Data Efficiency via Curating LLM-Driven Rating Systemscited by 29
  27. 2024Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sourcescited by 28
  28. 2022TabSynDex: A Universal Metric for Robust Evaluation of Synthetic Tabular Datacited by 27
  29. 2024Automated Data Curation for Robust Language Model Fine-Tuningcited by 27
  30. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  31. 2023People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detectioncited by 25
  32. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
  33. 2024RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Responsecited by 24
  34. 2024Distilling Large Language Models for Text-Attributed Graph Learningcited by 20
  35. 2023CoachLM: Automatic Instruction Revisions Improve the Data Quality in LLM Instruction Tuningcited by 19
  36. 2025Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinementcited by 18
  37. 2024Empowering Large Language Models for Textual Data Augmentationcited by 17
  38. 2024Synthetic Oversampling: Theory and A Practical Approach Using LLMs to Address Data Imbalancecited by 17

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  2. 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
  3. 2026CoAct: Co-Active LLM Preference Learning with Human-AI Synergycited by 3
  4. 2026Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Codecited by 1
  5. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  6. 2026Towards Efficient LLMs Annealing with Principled Sample Selectioncited by 0
  7. 2026SDO: Structure-Aware Data Organization for Efficient LLM Post-Trainingcited by 0
  8. 2026SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Datacited by 0
  9. 2026Relative Generalization Invariance of LLM Pretrainingcited by 0