# LLMs program using Reusable and Composable Abstract Representations

arXiv: https://arxiv.org/abs/2610.19238 · HTML: https://mlsanity.com/paper/2610.19238/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 36 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [Gemma 4 Technical Report](https://mlsanity.com/paper/2607.02770/) (2026)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [Extracting Latent Steering Vectors from Pretrained Language Models](https://mlsanity.com/paper/2205.05124/) (2022)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [RWKV-7 "Goose" with Expressive Dynamic State Evolution](https://mlsanity.com/paper/2503.14456/) (2025)
- [Transformers represent belief state geometry in their residual stream](https://mlsanity.com/paper/2405.15943/) (2024)
- [Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers](https://mlsanity.com/paper/2411.08745/) (2024)
- [Emergent Representations of Program Semantics in Language Models Trained on Programs](https://mlsanity.com/paper/2305.11169/) (2023)
- [IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators](https://mlsanity.com/paper/2403.03894/) (2024)
- [Vision-Language Models Create Cross-Modal Task Representations](https://mlsanity.com/paper/2410.22330/) (2024)
- [Do different prompting methods yield a common task representation in language models?](https://mlsanity.com/paper/2505.12075/) (2025)
- [KV Cache Steering for Controlling Frozen LLMs](https://mlsanity.com/paper/2507.08799/) (2025)
- [Differential syntactic and semantic encoding in LLMs](https://mlsanity.com/paper/2601.04765/) (2026)
- [Transformers learn factored representations](https://mlsanity.com/paper/2602.02385/) (2026)
- [Neural networks leverage nominally quantum and post-quantum representations](https://mlsanity.com/paper/2507.07432/) (2025)
- [Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study](https://mlsanity.com/paper/2310.16937/) (2023)
- [Understanding How CodeLLMs (Mis)Predict Types with Activation Steering](https://mlsanity.com/paper/2404.01903/) (2024)
- [Language Agnostic Code Embeddings](https://mlsanity.com/paper/2310.16803/) (2023)
- [Vector Arithmetic in Concept and Token Subspaces](https://mlsanity.com/paper/2511.18162/) (2025)
- [Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RL](https://mlsanity.com/paper/2604.20835/) (2026)
- [Steering Code LLMs with Activation Directions for Language and Library Control](https://mlsanity.com/paper/2603.23629/) (2026)

## Abstract (arXiv)

When writing code, the same computation can be expressed in different programming languages and through different implementation strategies. We show that the activations of LLMs capture this shared structure in reusable and composable abstract representations when they generate code. These representations emerge in the middle layers, are reusable across programming languages, and afford the model an impressive amount of generalization ability when generating code under various interventions. Using activation steering, we show that function, programming language, and implementation strategy can be independently manipulated. The independent interventions on language and function can be flexibly composed to control code generation, even in cases when the prompt contains no information about language and function. We additionally find that finetuning on examples of arbitrarily defined functions in one programming language transfers to other languages, consistent with the models reusing abstract features of code generation that are shared across different examples. Finally, to study how such abstract representations come to be, we implement a minimal model of shared, reusable, and composable structure in a synthetic setting, and show that transformers trained on such data learn representations that reuse these parts flexibly across contexts.