mlsanityData to 14 Oct 2026

Mastering the Craft of Data Synthesis for CodeLLMs

arXiv 2411.00005 ↗First posted 16 Oct 2024cs.SE cs.AIcited by 3

On the map: Synthetic data and self-improvement for LLMs → Code and instruction data synthesis

Abstract (arXiv)

Large language models (LLMs) have shown impressive performance in code understanding and generation, making coding tasks a key focus for researchers due to their practical applications and value as a testbed for LLM evaluation. Data synthesis and filtering techniques have been widely adopted and shown to be highly effective in this context. In this paper, we present a focused survey and taxonomy of these techniques, emphasizing recent advancements. We highlight key challenges, explore future research directions, and offer practical guidance for new researchers entering the field.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 69 of its 108 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

69 of 108 references.

References · 69

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  10. 2021Measuring Coding Challenge Competence With APPScited by 902
  11. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  12. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
38 more
  1. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  2. 2024A Survey on Large Language Models for Code Generationcited by 658
  3. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  4. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  5. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  6. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  7. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  8. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  9. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  10. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  11. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  12. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  13. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  14. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  15. 2024A Survey on Data Selection for Language Modelscited by 236
  16. 2017A Survey of Machine Learning for Big Code and Naturalnesscited by 221
  17. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  18. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  19. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  20. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  21. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 102
  22. 2023Human Feedback is not Gold Standardcited by 83
  23. 2022Language Models Can Teach Themselves to Program Bettercited by 78
  24. 2023Efficient Online Data Mixing For Language Model Pre-Trainingcited by 77
  25. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 77
  26. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  27. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 65
  28. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  29. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  30. 2023A Survey on Large Language Models for Software Engineeringcited by 59
  31. 2023Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuningcited by 58
  32. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 57
  33. 2024Synthesizing Text-to-SQL Data from Weak and Strong LLMscited by 54
  34. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 53
  35. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 52
  36. 2022Compilable Neural Code Generation with Compiler Feedbackcited by 47
  37. 2023ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systemscited by 45
  38. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  2. 2025LLMSynthor: Macro-Aligned Micro-Records Synthesis with Large Language Modelscited by 1
  3. 2026Not All Code Is Equal: A Data-Centric Study of Code Complexity and LLM Reasoningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  2. 2024Improving the Ability of Pre-trained Language Model by Imparting Large Language Model's Experience ↗
  3. 2024CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement
  4. 2025Data-efficient LLM Fine-tuning for Code Generation
  5. 2025Towards Advancing Code Generation with Large Language Models: A Research Roadmap
  6. 2024Case2Code: Scalable Synthetic Data for Code Generation
  7. 2024Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning
  8. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence