mlsanityData to 14 Oct 2026

In-Context Learning Amplifies a Latent Symbolic Circuit

arXiv 2609.36265 ↗First posted 28 Sep 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models can learn abstract rules from just a few in-context examples, but how their internal mechanisms activate as examples accumulate is not well understood. We trace a three-stage symbolic reasoning circuit (abstraction, induction, retrieval) across shot counts in three model families and find it is detectable and functional well before the model achieves high accuracy. Per-head causal contribution grows up to 8x from 1- to 10-shot, and cross-shot activation patching raises accuracy from 1% to 56% at 0-shot and 17% to 88% at 1-shot. Function vectors scaled and injected at 0-shot rescue accuracy up to 86%, largely substituting for the induction stage but depending critically on an intact downstream retrieval stage. The infrastructure for abstract rule-following is present in the weights before any demonstrations; in-context examples, function vectors, and related interventions appear to supply input to the same latent circuit.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 15 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

12 of 15 references.

References · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  3. 2023Steering Language Models With Activation Engineeringcited by 922
  4. 2022Transformers learn in-context by gradient descentcited by 732
  5. 2023In-Context Learning Creates Task Vectorscited by 312
  6. 2023Function Vectors in Large Language Modelscited by 308
  7. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  8. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
  9. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  10. 2025Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Modelscited by 31
  11. 2024LLMs as Models for Analogical Reasoningcited by 16
  12. 2026Causality neq Invariance: Function and Concept Vectors in LLMscited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Systematic Study of Small Language Models on Abstract Reasoning Tasks
  2. 2026Unveiling the Reasoning Process of Large Language Models
  3. 2024Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference
  4. 2026Revealing Algorithmic Deductive Circuits for Logical Reasoning
  5. 2026Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models
  6. 2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
  7. 2025Analogical Reasoning Inside Large Language Models: Concept Vectors and the Limits of Abstraction
  8. 2023Concept-aware Training Improves In-context Learning Ability of Language Models