# Transformers generalize differently from information stored in context vs in weights

arXiv: https://arxiv.org/abs/2210.05675 · HTML: https://mlsanity.com/paper/2210.05675/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022)
- [Black-Box Tuning for Language-Model-as-a-Service](https://mlsanity.com/paper/2201.03514/) (2022)
- [When redundancy is useful: A Bayesian approach to 'overinformative' referring expressions](https://mlsanity.com/paper/1903.08237/) (2019)
- [Distinguishing rule- and exemplar-based generalization in learning systems](https://mlsanity.com/paper/2110.04328/) (2021)

## Built on by (most cited first, 30 of 49)

- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [Generating Images with Multimodal Language Models](https://mlsanity.com/paper/2305.17216/) (2023)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [Uncovering mesa-optimization algorithms in Transformers](https://mlsanity.com/paper/2309.05858/) (2023)
- [In-Context Learning through the Bayesian Prism](https://mlsanity.com/paper/2306.04891/) (2023)
- [Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale](https://mlsanity.com/paper/2212.09095/) (2022)
- [In-Context Learning Learns Label Relationships but Is Not Conventional Learning](https://mlsanity.com/paper/2307.12375/) (2023)
- [Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations](https://mlsanity.com/paper/2305.13299/) (2023)
- [VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks](https://mlsanity.com/paper/2401.13649/) (2024)
- [HyperTuning: Toward Adapting Large Language Models without Back-propagation](https://mlsanity.com/paper/2211.12485/) (2022)
- [Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections](https://mlsanity.com/paper/2311.10678/) (2023)
- [On the generalization of language models from in-context learning and finetuning: a controlled study](https://mlsanity.com/paper/2505.00661/) (2025)
- [Generalization to New Sequential Decision Making Tasks with In-Context Learning](https://mlsanity.com/paper/2312.03801/) (2023)
- [The broader spectrum of in-context learning](https://mlsanity.com/paper/2412.03782/) (2024)
- [On the Out-Of-Distribution Generalization of Multimodal Large Language Models](https://mlsanity.com/paper/2402.06599/) (2024)
- [Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering](https://mlsanity.com/paper/2511.00617/) (2025)
- [A Closer Look at In-Context Learning under Distribution Shifts](https://mlsanity.com/paper/2305.16704/) (2023)
- [In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax](https://mlsanity.com/paper/2311.07811/) (2023)
- [Passive learning of active causal strategies in agents and language models](https://mlsanity.com/paper/2305.16183/) (2023)
- [When does compositional structure yield compositional generalization? A kernel theory](https://mlsanity.com/paper/2405.16391/) (2024)
- [LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History](https://mlsanity.com/paper/2402.18216/) (2024)
- [Toward Understanding In-context vs. In-weight Learning](https://mlsanity.com/paper/2410.23042/) (2024)
- [Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting](https://mlsanity.com/paper/2406.00053/) (2024)
- [Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiences](https://mlsanity.com/paper/2509.16189/) (2025)
- [Conceptual structure coheres in human cognition but not in large language models](https://mlsanity.com/paper/2304.02754/) (2023)
- [A Survey to Recent Progress Towards Understanding In-Context Learning](https://mlsanity.com/paper/2402.02212/) (2024)
- [Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer](https://mlsanity.com/paper/2506.01115/) (2025)
- [Ambiguity-Aware In-Context Learning with Large Language Models](https://mlsanity.com/paper/2309.07900/) (2023)
- [Breaking through the learning plateaus of in-context learning in Transformer](https://mlsanity.com/paper/2309.06054/) (2023)

## Abstract (arXiv)

Transformer models can use two fundamentally different kinds of information: information stored in weights during training, and information provided ``in-context'' at inference time. In this work, we show that transformers exhibit different inductive biases in how they represent and generalize from the information in these two sources. In particular, we characterize whether they generalize via parsimonious rules (rule-based generalization) or via direct comparison with observed examples (exemplar-based generalization). This is of important practical consequence, as it informs whether to encode information in weights or in context, depending on how we want models to use that information. In transformers trained on controlled stimuli, we find that generalization from weights is more rule-based whereas generalization from context is largely exemplar-based. In contrast, we find that in transformers pre-trained on natural language, in-context learning is significantly rule-based, with larger models showing more rule-basedness. We hypothesise that rule-based generalization from in-context information might be an emergent consequence of large-scale training on language, which has sparse rule-like structure. Using controlled stimuli, we verify that transformers pretrained on data containing sparse rule-like structure exhibit more rule-based generalization.