mlsanityData to 7 Oct 2026

Code Pretraining Improves Entity Tracking Abilities of Language Models

arXiv 2405.21068 ↗First posted 31 May 2024cs.CL cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

Recent work has provided indirect evidence that pretraining language models on code improves the ability of models to track state changes of discourse entities expressed in natural language. In this work, we systematically test this claim by comparing pairs of language models on their entity tracking performance. Critically, the pairs consist of base models and models trained on top of these base models with additional code data. We extend this analysis to additionally examine the effect of math training, another highly structured data type, and alignment tuning, an important step for enhancing the usability of models. We find clear evidence that models additionally trained on large amounts of code outperform the base models. On the other hand, we find no consistent benefit of additional math training or alignment tuning across various model families.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 22 of its 25 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

22 of 25 references.

References · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  10. 2023Scaling Data-Constrained Language Modelscited by 391
  11. 2023Llemma: An Open Language Model For Mathematicscited by 385
  12. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
10 more
  1. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  2. 2023Efficient Guided Generation for Large Language Modelscited by 192
  3. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  4. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  5. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  6. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  7. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  8. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  9. 2023Entity Tracking in Language Modelscited by 44
  10. 2019Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Textcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 14.

31 papers.

Sort:

By area:

  1. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 50
  2. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  3. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  4. 2024Representational Analysis of Binding in Language Modelscited by 27
  5. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  6. 2024A Notion of Complexity for Theory of Mind via Discrete World Modelscited by 16
  7. 2024EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Modelscited by 16
  8. 2025General Intelligence Requires Reward-based Pretrainingcited by 12
  9. 2025Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resourcescited by 10
  10. 2024Major Entity Identification: A Generalizable Alternative to Coreference Resolutioncited by 9
  11. 2024Chain and Causal Attention for Efficient Entity Trackingcited by 8
  12. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
19 more
  1. 2026Do Language Models Track Entities Across State Changes?cited by 5
  2. 2026A retrieval conditioned rebinding circuit for dynamic entity tracking in large language modelscited by 3
  3. 2026Domain-Aware Scaling Laws Uncover Data Synergycited by 2
  4. 2025MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Modelscited by 1
  5. 2026Can Large Language Models Generalize Procedures Across Representations?cited by 1
  6. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  7. 2025Large language models have learned to use languagecited by 0
  8. 2025Large language models are not about natural languagecited by 0
  9. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  10. 2026Are Language Models Models?cited by 0
  11. 2026A Family of LLMs Liberated from Static Vocabulariescited by 0
  12. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
  13. 2026You Can't Fight in Here! This is BBS!cited by 0
  14. 2026Brain Score Tracks Shared Properties of Languages: Evidence from Many Natural Languages and Structured Sequencescited by 0
  15. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0
  16. 2026Natural Identifiers for Privacy and Data Audits in Large Language Modelscited by 0
  17. 2026Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narrativescited by 0
  18. 2026In-Context Binding Capacity in Language Modelscited by 0
  19. 2026Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Priorcited by 0