mlsanityData to 14 Oct 2026

LLMs program using Reusable and Composable Abstract Representations

arXiv 2610.19238 ↗First posted 13 Oct 2026cs.LG cs.PLcited by 0

Not on the map. Why?

Abstract (arXiv)

When writing code, the same computation can be expressed in different programming languages and through different implementation strategies. We show that the activations of LLMs capture this shared structure in reusable and composable abstract representations when they generate code. These representations emerge in the middle layers, are reusable across programming languages, and afford the model an impressive amount of generalization ability when generating code under various interventions. Using activation steering, we show that function, programming language, and implementation strategy can be independently manipulated. The independent interventions on language and function can be flexibly composed to control code generation, even in cases when the prompt contains no information about language and function. We additionally find that finetuning on examples of arbitrarily defined functions in one programming language transfers to other languages, consistent with the models reusing abstract features of code generation that are shared across different examples. Finally, to study how such abstract representations come to be, we implement a minimal model of shared, reusable, and composable structure in a synthetic setting, and show that transformers trained on such data learn representations that reuse these parts flexibly across contexts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

31 of 36 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  6. 2023Steering Language Models With Activation Engineeringcited by 922
  7. 2015A Neural Algorithm of Artistic Stylecited by 483
  8. 2023In-Context Learning Creates Task Vectorscited by 312
  9. 2023Function Vectors in Large Language Modelscited by 308
  10. 2026Gemma 4 Technical Reportcited by 259
  11. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  12. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
19 more
  1. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  2. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  3. 2024Transformers represent belief state geometry in their residual streamcited by 77
  4. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 44
  5. 2023Emergent Representations of Program Semantics in Language Models Trained on Programscited by 30
  6. 2024IRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generatorscited by 23
  7. 2024Vision-Language Models Create Cross-Modal Task Representationscited by 15
  8. 2025Do different prompting methods yield a common task representation in language models?cited by 11
  9. 2025KV Cache Steering for Controlling Frozen LLMscited by 11
  10. 2026Differential syntactic and semantic encoding in LLMscited by 7
  11. 2026Transformers learn factored representationscited by 7
  12. 2025Neural networks leverage nominally quantum and post-quantum representationscited by 6
  13. 2023Cross-lingual Transfer in Programming Languages: An Extensive Empirical Studycited by 5
  14. 2024Understanding How CodeLLMs (Mis)Predict Types with Activation Steeringcited by 5
  15. 2023Language Agnostic Code Embeddingscited by 4
  16. 2025Vector Arithmetic in Concept and Token Subspacescited by 4
  17. 2026Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RLcited by 2
  18. 2026Steering Code LLMs with Activation Directions for Language and Library Controlcited by 1
  19. 2026Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Languagecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
  2. 2025On Code-Induced Reasoning in LLMs
  3. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
  4. 2026Detecting Functional Memorization in Code Language Models
  5. 2025Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking
  6. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
  7. 2025code_transformed: The Influence of Large Language Models on Code
  8. 2024Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations