# ICLR: In-Context Learning of Representations

arXiv: https://arxiv.org/abs/2501.00070 · HTML: https://mlsanity.com/paper/2501.00070/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 70 references that are in the map.
- First posted on arXiv 29 Dec 2024.

## Builds on (most cited first, 30 of 51)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [Transformers Learn Shortcuts to Automata](https://mlsanity.com/paper/2210.10749/) (2022)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [In-context Reinforcement Learning with Algorithm Distillation](https://mlsanity.com/paper/2210.14215/) (2022)
- [A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity](https://mlsanity.com/paper/2401.01967/) (2024)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Supervised Pretraining Can Learn In-Context Reinforcement Learning](https://mlsanity.com/paper/2306.14892/) (2023)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks](https://mlsanity.com/paper/2311.12786/) (2023)

## Built on by (most cited first, 30 of 50)

- [Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs](https://mlsanity.com/paper/2502.08640/) (2025)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering](https://mlsanity.com/paper/2511.00617/) (2025)
- [Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior](https://mlsanity.com/paper/2605.05115/) (2026)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time](https://mlsanity.com/paper/2510.01098/) (2025)
- [Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamics](https://mlsanity.com/paper/2511.04527/) (2025)
- [A Survey of Scaling in Large Language Model Reasoning](https://mlsanity.com/paper/2504.02181/) (2025)
- [Linear representations in language models can change dramatically over a conversation](https://mlsanity.com/paper/2601.20834/) (2026)
- [Context Structure Reshapes the Representational Geometry of Language Models](https://mlsanity.com/paper/2601.22364/) (2026)
- [Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls](https://mlsanity.com/paper/2510.00184/) (2025)
- [When Is Collective Intelligence a Lottery? Multi-Agent Scaling Laws for Memetic Drift in LLMs](https://mlsanity.com/paper/2603.24676/) (2026)
- [DETACH: Cross-domain Learning for Long-Horizon Tasks via Mixture of Disentangled Experts](https://mlsanity.com/paper/2508.07842/) (2025)
- [In-Context Representation Hijacking](https://mlsanity.com/paper/2512.03771/) (2025)
- [In-Context Algebra](https://mlsanity.com/paper/2512.16902/) (2025)
- [Harmonic Loss Trains Interpretable AI Models](https://mlsanity.com/paper/2502.01628/) (2025)
- [Rapid Word Learning Through Meta In-Context Learning](https://mlsanity.com/paper/2502.14791/) (2025)
- [Provable In-Context Learning of Nonlinear Regression with Transformers](https://mlsanity.com/paper/2507.20443/) (2025)
- [On Entity Identification in Language Models](https://mlsanity.com/paper/2506.02701/) (2025)
- [Provable Low-Frequency Bias of In-Context Learning of Representations](https://mlsanity.com/paper/2507.13540/) (2025)
- [Beyond External Monitors: Enhancing Transparency of Large Language Models for Easier Monitoring](https://mlsanity.com/paper/2502.05242/) (2025)
- [A Markov Categorical Framework for Language Modeling](https://mlsanity.com/paper/2507.19247/) (2025)
- [Review of Hallucination Understanding in Large Language and Vision Models](https://mlsanity.com/paper/2510.00034/) (2025)
- [Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings](https://mlsanity.com/paper/2510.19694/) (2025)
- [Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agents](https://mlsanity.com/paper/2602.05810/) (2026)
- [Reasoning aligns language models to human cognition](https://mlsanity.com/paper/2602.08693/) (2026)
- [Polar probe linearly decodes semantic structures from LLMs](https://mlsanity.com/paper/2605.14125/) (2026)
- [Large language models reorganize representational geometry during in-context learning](https://mlsanity.com/paper/2605.28854/) (2026)
- [From Memories to Maps: Mechanisms of In-Context Reinforcement Learning in Transformers](https://mlsanity.com/paper/2506.19686/) (2025)
- [A circuit for predicting hierarchical structure in-context in Large Language Models](https://mlsanity.com/paper/2509.21534/) (2025)

## Abstract (arXiv)

Recent work has demonstrated that semantics specified by pretraining data influence how representations of different concepts are organized in a large language model (LLM). However, given the open-ended nature of LLMs, e.g., their ability to in-context learn, we can ask whether models alter these pretraining semantics to adopt alternative, context-specified ones. Specifically, if we provide in-context exemplars wherein a concept plays a different role than what the pretraining data suggests, do models reorganize their representations in accordance with these novel semantics? To answer this question, we take inspiration from the theory of conceptual role semantics and define a toy "graph tracing" task wherein the nodes of the graph are referenced via concepts seen during training (e.g., apple, bird, etc.) and the connectivity of the graph is defined via some predefined structure (e.g., a square grid). Given exemplars that indicate traces of random walks on the graph, we analyze intermediate representations of the model and find that as the amount of context is scaled, there is a sudden re-organization from pretrained semantic representations to in-context representations aligned with the graph structure. Further, we find that when reference concepts have correlations in their semantics (e.g., Monday, Tuesday, etc.), the context-specified graph structure is still present in the representations, but is unable to dominate the pretrained structure. To explain these results, we analogize our task to energy minimization for a predefined graph topology, providing evidence towards an implicit optimization process to infer context-specified semantics. Overall, our findings indicate scaling context-size can flexibly re-organize model representations, possibly unlocking novel capabilities.