# Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning

arXiv: https://arxiv.org/abs/2405.20535 · HTML: https://mlsanity.com/paper/2405.20535/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 30 May 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 34)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024)
- [Language Models of Code are Few-Shot Commonsense Learners](https://mlsanity.com/paper/2210.07128/) (2022)
- [Holistic Evaluation of Language Models](https://mlsanity.com/paper/2211.09110/) (2022)
- [Explanations from Large Language Models Make Small Reasoners Better](https://mlsanity.com/paper/2210.06726/) (2022)
- [Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization](https://mlsanity.com/paper/2302.08081/) (2023)
- [AlpaCare:Instruction-tuned Large Language Models for Medical Application](https://mlsanity.com/paper/2310.14558/) (2023)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [At Which Training Stage Does Code Data Help LLMs Reasoning?](https://mlsanity.com/paper/2309.16298/) (2023)
- [A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law](https://mlsanity.com/paper/2405.01769/) (2024)

## Built on by (most cited first, 30 of 31)

- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [To Code, or Not To Code? Exploring Impact of Code in Pre-training](https://mlsanity.com/paper/2408.10914/) (2024)
- [Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs](https://mlsanity.com/paper/2502.19411/) (2025)
- [Evaluating Language Models as Synthetic Data Generators](https://mlsanity.com/paper/2412.03679/) (2024)
- [Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping](https://mlsanity.com/paper/2410.16232/) (2024)
- [Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training](https://mlsanity.com/paper/2509.26625/) (2025)
- [EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models](https://mlsanity.com/paper/2409.17892/) (2024)
- [Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality](https://mlsanity.com/paper/2506.14681/) (2025)
- [SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation](https://mlsanity.com/paper/2507.09108/) (2025)
- [Coding Agents are Effective Long-Context Processors](https://mlsanity.com/paper/2603.20432/) (2026)
- [CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning](https://mlsanity.com/paper/2410.02229/) (2024)
- [Only-IF:Revealing the Decisive Effect of Instruction Diversity on Generalization](https://mlsanity.com/paper/2410.04717/) (2024)
- [Automatic Textbook Formalization](https://mlsanity.com/paper/2604.03071/) (2026)
- [Thinking with Knowledge Graphs: Enhancing LLM Reasoning Through Structured Data](https://mlsanity.com/paper/2412.10654/) (2024)
- [Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?](https://mlsanity.com/paper/2505.12766/) (2025)
- [When Style Breaks Safety: Defending LLMs Against Superficial Style Alignment](https://mlsanity.com/paper/2506.07452/) (2025)
- [Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks](https://mlsanity.com/paper/2510.01232/) (2025)
- [On Code-Induced Reasoning in LLMs](https://mlsanity.com/paper/2509.21499/) (2025)
- [Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning](https://mlsanity.com/paper/2506.10378/) (2025)
- [CoinMath: Harnessing the Power of Coding Instruction for Math LLMs](https://mlsanity.com/paper/2412.11699/) (2024)
- [Multi Agent based Medical Assistant for Edge Devices](https://mlsanity.com/paper/2503.05397/) (2025)
- [Latent Traits and Cross-Task Transfer: Deconstructing Dataset Interactions in LLM Fine-tuning](https://mlsanity.com/paper/2509.13624/) (2025)
- [From Questions to Queries: An AI-powered Multi-Agent Framework for Spatial Text-to-SQL](https://mlsanity.com/paper/2510.21045/) (2025)
- [A Latent Variable Framework for Scaling Laws in Large Language Models](https://mlsanity.com/paper/2512.06553/) (2025)
- [R1-Ranker: Teaching LLM Rankers to Reason](https://mlsanity.com/paper/2506.21638/) (2025)
- [Not All Code Is Equal: A Data-Centric Study of Code Complexity and LLM Reasoning](https://mlsanity.com/paper/2601.21894/) (2026)
- [Can Large Language Models Generalize Procedures Across Representations?](https://mlsanity.com/paper/2602.03542/) (2026)
- [Reforming the Mechanism: Editing Reasoning Patterns in LLMs with Circuit Reshaping](https://mlsanity.com/paper/2603.06923/) (2026)
- [FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Models](https://mlsanity.com/paper/2407.01046/) (2024)
- [The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models](https://mlsanity.com/paper/2602.01161/) (2026)

## Abstract (arXiv)

Instruction Fine-Tuning (IFT) significantly enhances the zero-shot capabilities of pretrained Large Language Models (LLMs). While coding data is known to boost LLM reasoning abilities during pretraining, its role in activating internal reasoning capacities during IFT remains understudied. This paper investigates a key question: How does coding data impact LLMs' reasoning capacities during IFT stage? To explore this, we thoroughly examine the impact of coding data across different coding data proportions, model families, sizes, and reasoning domains, from various perspectives. Specifically, we create three IFT datasets with increasing coding data proportions, fine-tune six LLM backbones across different families and scales on these datasets, evaluate the tuned models' performance across twelve tasks in three reasoning domains, and analyze the outcomes from three broad-to-granular perspectives: overall, domain-level, and task-specific. Our holistic analysis provides valuable insights into each perspective. First, coding data tuning enhances the overall reasoning capabilities of LLMs across different model families and scales. Moreover, while the impact of coding data varies by domain, it shows consistent trends within each domain across different model families and scales. Additionally, coding data generally provides comparable task-specific benefits across model families, with optimal proportions in IFT datasets being task-dependent.