# Understanding How CodeLLMs (Mis)Predict Types with Activation Steering arXiv: https://arxiv.org/abs/2404.01903 ยท HTML: https://mlsanity.com/paper/2404.01903/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability. - Builds on 24 of its 65 references that are in the map. - First posted on arXiv 2 Apr 2024. ## Placed in - Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/) ## Builds on (most cited first, 24 of 24) - [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023) - [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023) - [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024) - [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023) - [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022) - [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022) - [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022) - [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023) - [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024) - [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022) - [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023) - [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020) - [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023) - [If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents](https://mlsanity.com/paper/2401.00812/) (2024) - [Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions](https://mlsanity.com/paper/2212.10561/) (2022) - [Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction](https://mlsanity.com/paper/2105.06965/) (2021) - [LambdaNet: Probabilistic Type Inference using Graph Neural Networks](https://mlsanity.com/paper/2005.02161/) (2020) - [Deploying and Evaluating LLMs to Program Service Mobile Robots](https://mlsanity.com/paper/2311.11183/) (2023) - [Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models](https://mlsanity.com/paper/2312.10091/) (2023) - [OptTyper: Probabilistic Type Inference by Optimising Logical and Natural Constraints](https://mlsanity.com/paper/2004.00348/) (2020) - [Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates](https://mlsanity.com/paper/2402.05980/) (2024) - [ManyTypes4Py: A Benchmark Python Dataset for Machine Learning-based Type Inference](https://mlsanity.com/paper/2104.04706/) (2021) - [Do Machine Learning Models Produce TypeScript Types That Type Check?](https://mlsanity.com/paper/2302.12163/) (2023) - [Solver-based Gradual Type Migration](https://mlsanity.com/paper/2109.05049/) (2021) ## Built on by (most cited first, 5 of 5) - [Steering LLMs for Formal Theorem Proving](https://mlsanity.com/paper/2502.15507/) (2025) - [Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models](https://mlsanity.com/paper/2602.02462/) (2026) - [TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models](https://mlsanity.com/paper/2607.08339/) (2026) - ["As a Language Model...": Chat Template Switches LLM Self-Referential Voice and Activation Steering Reproduces It](https://mlsanity.com/paper/2609.25021/) (2026) - [LLMs program using Reusable and Composable Abstract Representations](https://mlsanity.com/paper/2610.19238/) (2026) ## Abstract (arXiv) Large Language Models (LLMs) are widely used by software engineers for programming tasks. However, research shows that LLMs often lack a deep understanding of program semantics. Even minor changes to syntax, such as renaming variables, can significantly degrade performance across various tasks. In this work, we examine the task of type prediction: given a partially typed program, can a model predict a missing type annotations such that the resulting program is more typed? We construct a dataset of adversarial examples where models initially predict the correct types, but begin to fail after semantically irrelevant edits. This is problematic, as models should ideally generalize across different syntactic forms of semantically equivalent code. This lack of robustness suggests that models may have a shallow understanding of code semantics. Despite this, we provide evidence that LLMs do, in fact, learn robust mechanisms for type prediction-though these mechanisms often fail to activate in adversarial scenarios. By using activation steering, a method that manipulates a model's internal activations to guide it toward using latent knowledge, we restore accurate predictions on adversarial inputs. We show that steering successfully activates a type prediction mechanism that is shared by both Python and TypeScript, and is more effective than prompting with in-context examples. Across five different models, our comprehensive evaluation demonstrates that LLMs can learn generalizable representations of code semantics that transfer across programming languages.