mlsanityData to 7 Oct 2026

How code helps different tasks? A decompositional lens on LLM post-training

arXiv 2609.33845 ↗First posted 27 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Evaluating code data as a single corpus can obscure which types of code data benefit which models and downstream tasks. Effective data selection requires understanding both the benefits of individual categories and whether these benefits persist when categories are combined. We introduce a decompositional lens for studying these effects in LLM post-training. We first decompose an execution-verified code corpus into interpretable categories based on the computational patterns of its solutions. Through controlled fine-tuning experiments, we compare individual categories with a balanced mixture across instruction-tuned models on question answering, mathematics, and code generation. The resulting response maps reveal recurring gains in average question-answering performance, while the same category can improve one model or task and degrade another. The best-performing category also varies with the starting model and target task. We then compose compact mixtures guided by these results and examine whether benefits observed in individual categories persist under joint training. On selected model--task pairs, mixtures whose constituents each improve the target task outperform both their best constituent and full-corpus training while using roughly 10--15% of the full corpus. These exploratory findings illustrate a less is more pattern and highlight how the value of code data in post training depends on which categories are combined for which model and task.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 33 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

32 of 33 references.

References · 32

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  10. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  11. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  12. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
20 more
  1. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  2. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  3. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  4. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  5. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  6. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  7. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  8. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 108
  9. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  10. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 64
  11. 2025CodeI/O: Condensing Reasoning Patterns via Code Input-Output Predictioncited by 54
  12. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  13. 2024Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoningcited by 31
  14. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  15. 2025Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoningcited by 10
  16. 2025Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functionscited by 7
  17. 2025DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collectionscited by 6
  18. 2025Data-efficient LLM Fine-tuning for Code Generationcited by 5
  19. 2025On Code-Induced Reasoning in LLMscited by 5
  20. 2026Not All Code Is Equal: A Data-Centric Study of Code Complexity and LLM Reasoningcited by 1

Built on by

Not cited yet.