mlsanityData to 7 Oct 2026

CDS: Knowledge Component-Driven Data Synthesis Guided by Cognitive Diagnosis Theory

arXiv 2501.07674 ↗First posted 13 Jan 2025cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → Synthetic reasoning data and bootstrapping

Abstract (arXiv)

Large Language Models (LLMs) have achieved significant advancements, but the increasing complexity of tasks and higher performance demands highlight the need for continuous improvement. Some approaches utilize synthetic data generated by advanced LLMs based on evaluation results to train models. However, conventional evaluation methods fail to provide detailed, fine-grained profiles of LLMs, limiting their guidance for data synthesis. In this paper, we introduce the Cognitive Diagnostic Synthesis (CDS) method, which incorporates a diagnostic process inspired by Cognitive Diagnosis Theory (CDT) to refine evaluation results and characterize model profiles at the knowledge component level. Based on these diagnostics, we propose two diagnosis-synthesis strategies for weakness-targeted data synthesis. Additionally, we present an enhanced data augmentation and selection pipeline to improve the quality and diversity of synthesized data. Our experiments with several open-source models show significant improvements across multiple benchmarks, achieving up to 6.00% improvement in code generation, 13.10% in mathematical reasoning, and 5.43% in academic exams. Code and data are available on GitHub.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 29 of its 52 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

29 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning4 Mar

Other references · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  10. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  11. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  12. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
16 more
  1. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  2. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  3. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  4. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  5. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 108
  6. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  7. 2023Learning From Mistakes Makes LLM Better Reasonercited by 97
  8. 2019Neural Cognitive Diagnosis for Intelligent Education Systemscited by 75
  9. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  10. 2023Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuningcited by 56
  11. 2024MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Datacited by 43
  12. 2024In-Context Principle Learning from Mistakescited by 42
  13. 2023LogiCoT: Logical Chain-of-Thought Instruction-Tuningcited by 31
  14. 2024LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvementcited by 11
  15. 2024Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Taskscited by 5
  16. 2024CEM: A Data-Efficient Method for Large Language Models to Continue Evolving From Mistakescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond ID Embeddings: Process-Grounded Language Modeling for Cognitive Diagnosis
  2. 2026Scalable Text-Embedding-informed Cognitive Diagnosis of Large Language Models
  3. 2025More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning
  4. 2026LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB
  5. 2025Harnessing LLM for Noise-Robust Cognitive Diagnosis in Web-Based Intelligent Education Systems
  6. 2025Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work
  7. 2024Diagnosing and Remedying Knowledge Deficiencies in LLMs via Label-free Curricular Meaningful Learning
  8. 2025Test Case Generation from Bug Reports via Large Language Models: A Cognitive Layered Evaluation Framework