mlsanityData to 14 Oct 2026

To Code, or Not To Code? Exploring Impact of Code in Pre-training

arXiv 2408.10914 ↗First posted 20 Aug 2024cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

Including code in the pre-training data mixture, even for models not specifically designed for code, has become a common practice in LLMs pre-training. While there has been anecdotal consensus among practitioners that code data plays a vital role in general LLMs' performance, there is only limited work analyzing the precise impact of code on non-code tasks. In this work, we systematically investigate the impact of code data on general performance. We ask "what is the impact of code data used in pre-training on a large variety of downstream tasks beyond code generation". We conduct extensive ablations and evaluate across a broad range of natural language reasoning tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win-rates for models with sizes ranging from 470M to 2.8B parameters. Across settings, we find a consistent results that code is a critical building block for generalization far beyond coding tasks and improvements to code quality have an outsized impact across all tasks. In particular, compared to text-only pre-training, the addition of code results in up to relative increase of 8.2% in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement in generative win-rates, and a 12x boost in code performance respectively. Our work suggests investments in code quality and preserving code during pre-training have positive impacts.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 45 of its 67 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

45 of 67 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
33 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  2. 2020GLU Variants Improve Transformercited by 1,804
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  5. 2023StarCoder: may the source be with you!cited by 799
  6. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  10. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  11. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  12. 2023Textbooks Are All You Needcited by 569
  13. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  14. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  15. 2023Scaling Data-Constrained Language Modelscited by 403
  16. 2018QuAC : Question Answering in Contextcited by 381
  17. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  18. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  19. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  20. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  21. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  22. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  23. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  24. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  25. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
  26. 2024Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoningcited by 31
  27. 2024Code Pretraining Improves Entity Tracking Abilities of Language Modelscited by 31
  28. 2018Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehensioncited by 23
  29. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  30. 2024Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruningcited by 10
  31. 2022Scalable Training of Language Models using JAX pjit and TPUv4cited by 9
  32. 2022Conditional set generation using Seq2seq modelscited by 7
  33. 2024LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectivescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 23; 2026: 17.

47 papers.

Sort:

By area:

  1. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  2. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 76
  3. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 51
  4. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  5. 2024Procedural Knowledge in Pretraining Drives Reasoning in Large Language Modelscited by 41
  6. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 38
  7. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 37
  8. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 31
  9. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  10. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  11. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 20
  12. 2025IPO: Your Language Model is Secretly a Preference Classifiercited by 17
35 more
  1. 2025Could Thinking Multilingually Empower LLM Reasoning?cited by 16
  2. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 15
  3. 2025General Intelligence Requires Reward-based Pretrainingcited by 12
  4. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 12
  5. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  6. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  7. 2024CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoningcited by 10
  8. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  9. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  10. 2025ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Modelscited by 9
  11. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
  12. 2024FLARE: Faithful Logic-Aided Reasoning and Explorationcited by 7
  13. 2024FastDraft: How to Train Your Draftcited by 6
  14. 2024Thinking with Knowledge Graphs: Enhancing LLM Reasoning Through Structured Datacited by 6
  15. 2024Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepalicited by 5
  16. 2025How Programming Concepts and Neurons Are Shared in Code Language Modelscited by 5
  17. 2025Toward a Theory of Generalizability in LLM Mechanistic Interpretability Researchcited by 5
  18. 2026Will it Merge? On The Causes of Model Mergeabilitycited by 4
  19. 2025GPS: General Per-Sample Promptercited by 3
  20. 2025Grow Up and Merge: Scaling Strategies for Efficient Language Adaptationcited by 3
  21. 2025Is Compression Really Linear with Code Intelligence?cited by 2
  22. 2026Can Coding Agents Be General Agents?cited by 2
  23. 2026Domain-Aware Scaling Laws Uncover Data Synergycited by 2
  24. 2026Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Taskscited by 2
  25. 2026Can Large Language Models Generalize Procedures Across Representations?cited by 1
  26. 2026Knowledge Model Prompting Increases LLM Performance on Planning Taskscited by 1
  27. 2026MaD-Mix: Multi-Modal Data Mixtures via Latent Space Coupling for Vision-Language Model Trainingcited by 1
  28. 2026EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Trainingcited by 1
  29. 2026Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Codecited by 1
  30. 2026What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Codecited by 1
  31. 2025MALBO: Optimizing LLM-Based Multi-Agent Teams via Multi-Objective Bayesian Optimizationcited by 0
  32. 2026Natural Identifiers for Privacy and Data Audits in Large Language Modelscited by 0
  33. 2026Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narrativescited by 0
  34. 2026How code helps different tasks? A decompositional lens on LLM post-trainingcited by 0
  35. 2026Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Modelscited by 0