mlsanityData to 7 Oct 2026

How Does Code Pretraining Affect Language Model Task Performance?

arXiv 2409.04556 ↗First posted 6 Sep 2024cs.CL cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code. Aside from aiding programming-related tasks, anecdotal evidence suggests that including code in pretraining corpora may improve performance on other, unrelated tasks, yet to date no work has been able to establish a causal connection by controlling between language and code data. Here we do just this. We pretrain language models on datasets which interleave natural language and code in two different settings: additive, in which the total volume of data seen during pretraining is held constant; and competitive, in which the volume of language data is held constant. We study how the pretraining mixture affects performance on (a) a diverse collection of tasks included in the BigBench benchmark, and (b) compositionality, measured by generalization accuracy on semantic parsing and syntactic transformations. We find that pretraining on higher proportions of code improves performance on compositional tasks involving structured output (like semantic parsing), and mathematics. Conversely, increase code mixture can harm performance on other tasks, including on tasks that requires sensitivity to linguistic structure such as syntax or morphology, and tasks measuring real-world knowledge.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 6 Sep 2024.

Builds on

34 of 44 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  11. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  12. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
22 more
  1. 2020Unsupervised Translation of Programming Languagescited by 262
  2. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  3. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  4. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  5. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  6. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  7. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  8. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  9. 2021Examining Zero-Shot Vulnerability Repair with Large Language Modelscited by 105
  10. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  11. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  12. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  13. 2024CogBench: a large language model walks into a psychology labcited by 60
  14. 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
  15. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 37
  16. 2023Injecting structural hints: Using language models to study inductive biases in language learningcited by 33
  17. 2024Code Pretraining Improves Entity Tracking Abilities of Language Modelscited by 31
  18. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  19. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntaxcited by 22
  20. 2022Beyond the C: Retargetable Decompilation using Neural Machine Translationcited by 15
  21. 2023Exploring the Curious Case of Code Promptscited by 6
  22. 2024Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 16; 2026: 11.

28 papers.

Sort:

By area:

  1. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  2. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  3. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 36
  4. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 19
  5. 2025IPO: Your Language Model is Secretly a Preference Classifiercited by 17
  6. 2025General Intelligence Requires Reward-based Pretrainingcited by 12
  7. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  8. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  9. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  10. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
  11. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  12. 2025Don't throw the baby out with the bathwater: How and why deep learning for ARCcited by 7
16 more
  1. 2026CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Modelscited by 5
  2. 2025Trillion 7B Technical Reportcited by 4
  3. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  4. 2025Programming by Backprop: An Instruction is Worth 100 Examples When Finetuning LLMscited by 2
  5. 2026Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Taskscited by 2
  6. 2025Privacy-Preserving Dataset Combinationcited by 1
  7. 2025Empirical Study of Code Large Language Models for Binary Security Patch Detectioncited by 1
  8. 2026Can Large Language Models Generalize Procedures Across Representations?cited by 1
  9. 2024Uncovering Autoregressive LLM Knowledge of Thematic Fit in Event Representationcited by 0
  10. 2026CrossTrace: A Cross-Domain Dataset of Grounded Scientific Reasoning Traces for Hypothesis Generationcited by 0
  11. 2026Natural Identifiers for Privacy and Data Audits in Large Language Modelscited by 0
  12. 2026From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inferencecited by 0
  13. 2026Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?cited by 0
  14. 2026Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narrativescited by 0
  15. 2026How code helps different tasks? A decompositional lens on LLM post-trainingcited by 0
  16. 2026Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Priorcited by 0