mlsanityData to 7 Oct 2026

At Which Training Stage Does Code Data Help LLMs Reasoning?

arXiv 2309.16298 ↗First posted 28 Sep 2023cs.CLcited by 82

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) have exhibited remarkable reasoning capabilities and become the foundation of language technologies. Inspired by the great success of code data in training LLMs, we naturally wonder at which training stage introducing code data can really help LLMs reasoning. To this end, this paper systematically explores the impact of code data on LLMs at different stages. Concretely, we introduce the code data at the pre-training stage, instruction-tuning stage, and both of them, respectively. Then, the reasoning capability of LLMs is comprehensively and fairly evaluated via six reasoning tasks in five domains. We critically analyze the experimental results and provide conclusions with insights. First, pre-training LLMs with the mixture of code and text can significantly enhance LLMs' general reasoning capability almost without negative transfer on other tasks. Besides, at the instruction-tuning stage, code data endows LLMs the task-specific reasoning capability. Moreover, the dynamic mixing strategy of code and text data assists LLMs to learn reasoning capability step-by-step during training. These insights deepen the understanding of LLMs regarding reasoning ability for their application, such as scientific question answering, legal support, etc. The source code and model parameters are released at the link:~urlhttps://github.com/yingweima2022/CodeLLM.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 31 of its 44 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

31 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Code Llama: Open Foundation Models for Code24 Aug
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  3. 2022E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning16 Mar

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
16 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  3. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  4. 2023Instruction Tuning with GPT-4cited by 676
  5. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  6. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  7. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  8. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 115
  9. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
  10. 2020OCNLI: Original Chinese Natural Language Inferencecited by 86
  11. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 80
  12. 2021CoSQA: 20,000+ Web Queries for Code Search and Question Answeringcited by 74
  13. 2023CodeIE: Large Code Generation Models are Better Few-Shot Information Extractorscited by 67
  14. 2020CPM: A Large-scale Generative Chinese Pre-trained Language Modelcited by 60
  15. 2022PanGu-Coder: Program Synthesis with Function-Level Language Modelingcited by 47
  16. 2021OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 36; 2025: 32; 2026: 13.

82 papers.

Sort:

By area:

  1. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  2. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
  3. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  4. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  5. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  6. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  7. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  8. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  9. 2025CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generationcited by 65
  10. 2024Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvementcited by 57
  11. 2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
  12. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
38 more
  1. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  2. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  3. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  4. 2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
  5. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 32
  6. 2024Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoningcited by 31
  7. 2024Code Pretraining Improves Entity Tracking Abilities of Language Modelscited by 31
  8. 2024Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototypingcited by 29
  9. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
  10. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  11. 2025Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Computecited by 25
  12. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
  13. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  14. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  15. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  16. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  17. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 19
  18. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  19. 2024Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?cited by 17
  20. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 17
  21. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  22. 2025Token Weighting for Long-Range Language Modelingcited by 14
  23. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  24. 2025Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuningcited by 14
  25. 2024Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?cited by 13
  26. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  27. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 11
  28. 2026Coding Agents are Effective Long-Context Processorscited by 11
  29. 2024CodeTaxo: Enhancing Taxonomy Expansion with Limited Examples via Code Language Promptscited by 10
  30. 2024CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoningcited by 10
  31. 2024LLMs as Continuous Learners: Improving the Reproduction of Defective Code in Software Issuescited by 10
  32. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  33. 2024An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasetscited by 8
  34. 2025R-TOFU: Unlearning in Large Reasoning Modelscited by 8
  35. 2024Are LLMs classical or nonmonotonic reasoners? Lessons from genericscited by 7
  36. 2025ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Modelscited by 7
  37. 2025Don't throw the baby out with the bathwater: How and why deep learning for ARCcited by 7
  38. 2025Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levelscited by 7

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models ↗
  2. 2025An Empirical Study of Reasoning Steps in Thinking Code LLMs
  3. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training
  4. 2024Eliciting Better Multilingual Structured Reasoning from LLMs through Code
  5. 2025Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMs
  6. 2025Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
  7. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  8. 2024Can LLMs Reason in the Wild with Programs?