# Language Models Struggle to Use Representations Learned In-Context

arXiv: https://arxiv.org/abs/2602.04212 · HTML: https://mlsanity.com/paper/2602.04212/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 15 of 15)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Thought Anchors: Which LLM Reasoning Steps Matter?](https://mlsanity.com/paper/2506.19143/) (2025)
- [The broader spectrum of in-context learning](https://mlsanity.com/paper/2412.03782/) (2024)
- [Priors in Time: Missing Inductive Biases for Language Model Interpretability](https://mlsanity.com/paper/2511.01836/) (2025)
- [In-Context Learning Strategies Emerge Rationally](https://mlsanity.com/paper/2506.17859/) (2025)
- [Thought Branches: Interpreting LLM Reasoning Requires Resampling](https://mlsanity.com/paper/2510.27484/) (2025)
- [Do different prompting methods yield a common task representation in language models?](https://mlsanity.com/paper/2505.12075/) (2025)
- [What Does it Mean for a Neural Network to Learn a "World Model"?](https://mlsanity.com/paper/2507.21513/) (2025)
- [In-Context Representation Hijacking](https://mlsanity.com/paper/2512.03771/) (2025)
- [Better World Models Can Lead to Better Post-Training Performance](https://mlsanity.com/paper/2512.03400/) (2025)

## Built on by (most cited first, 6 of 6)

- [Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior](https://mlsanity.com/paper/2605.05115/) (2026)
- [Instructions Shape Production of Language, not Processing](https://mlsanity.com/paper/2605.11206/) (2026)
- [Causal Interventions on Continuous Variables: A Case Study on Verb Bias in Steering Vectors for In-Context Learning](https://mlsanity.com/paper/2605.29971/) (2026)
- [Context Is King: How In-Context Specification Shapes the Geometry of Concepts](https://mlsanity.com/paper/2607.24425/) (2026)
- [Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMs](https://mlsanity.com/paper/2609.24352/) (2026)
- [Decodable In-Context State and Model Output Across Training](https://mlsanity.com/paper/2609.31401/) (2026)

## Abstract (arXiv)

Though large language models (LLMs) have enabled great success across a wide variety of tasks, they still appear to fall short of one of the loftier goals of artificial intelligence research: creating an artificial system that can adapt its behavior to radically new contexts upon deployment. One important step towards this goal is to create systems that can induce rich representations of data that are seen in-context, and then flexibly deploy these representations to accomplish goals. Recently, Park et al. (2024) demonstrated that current LLMs are indeed capable of inducing such representation from context (i.e., in-context representation learning). The present study investigates whether LLMs can use these representations to complete simple downstream tasks. We first assess whether open-weights LLMs can use in-context representations for next-token prediction, and then probe models using a novel task, adaptive world modeling. In both tasks, we find evidence that open-weights LLMs struggle to deploy representations of novel semantics that are defined in-context, even if they encode these semantics in their latent representations. Furthermore, we assess closed-source, state-of-the-art reasoning models on the adaptive world modeling task, demonstrating that even the most performant LLMs cannot reliably leverage novel patterns presented in-context. Overall, this work seeks to inspire novel methods for encouraging models to not only encode information presented in-context, but to do so in a manner that supports flexible deployment of this information.