# How Do In-Context Examples Affect Compositional Generalization?

arXiv: https://arxiv.org/abs/2305.04835 · HTML: https://mlsanity.com/paper/2305.04835/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 8 May 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 39)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [Active Example Selection for In-Context Learning](https://mlsanity.com/paper/2211.04486/) (2022)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [Language Models of Code are Few-Shot Commonsense Learners](https://mlsanity.com/paper/2210.07128/) (2022)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?](https://mlsanity.com/paper/2010.12725/) (2020)
- [Diverse Demonstrations Improve In-context Compositional Generalization](https://mlsanity.com/paper/2212.06800/) (2022)
- [The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers](https://mlsanity.com/paper/2108.12284/) (2021)
- [Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures](https://mlsanity.com/paper/2007.08970/) (2020)
- [Compositional Semantic Parsing with Large Language Models](https://mlsanity.com/paper/2209.15003/) (2022)
- [Compositional Generalization via Neural-Symbolic Stack Machines](https://mlsanity.com/paper/2008.06662/) (2020)
- [Span-based Semantic Parsing for Compositional Generalization](https://mlsanity.com/paper/2009.06040/) (2020)
- [Learning to Recombine and Resample Data for Compositional Generalization](https://mlsanity.com/paper/2010.03706/) (2020)
- [Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations](https://mlsanity.com/paper/2104.07478/) (2021)
- [Compositional Generalization by Learning Analytical Expressions](https://mlsanity.com/paper/2006.10627/) (2020)

## Built on by (most cited first, 30 of 51)

- [Agent AI: Surveying the Horizons of Multimodal Interaction](https://mlsanity.com/paper/2401.03568/) (2024)
- [Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples](https://mlsanity.com/paper/2305.15269/) (2023)
- [In-Context Learning with Iterative Demonstration Selection](https://mlsanity.com/paper/2310.09881/) (2023)
- [Skill-Based Few-Shot Selection for In-Context Learning](https://mlsanity.com/paper/2305.14210/) (2023)
- [Why Larger Language Models Do In-context Learning Differently?](https://mlsanity.com/paper/2405.19592/) (2024)
- [Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability](https://mlsanity.com/paper/2407.15720/) (2024)
- [Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?](https://mlsanity.com/paper/2311.18021/) (2023)
- [Instruct Me More! Random Prompting for Visual In-Context Learning](https://mlsanity.com/paper/2311.03648/) (2023)
- [Understanding In-Context Learning from Repetitions](https://mlsanity.com/paper/2310.00297/) (2023)
- [Evaluating Morphological Compositional Generalization in Large Language Models](https://mlsanity.com/paper/2410.12656/) (2024)
- [How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?](https://mlsanity.com/paper/2404.12866/) (2024)
- [Attention as a Hypernetwork](https://mlsanity.com/paper/2406.05816/) (2024)
- [Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Models](https://mlsanity.com/paper/2402.15637/) (2024)
- [MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofs](https://mlsanity.com/paper/2410.13502/) (2024)
- [STAND-Guard: A Small Task-Adaptive Content Moderation Model](https://mlsanity.com/paper/2411.05214/) (2024)
- [Task Generalization With AutoRegressive Compositional Structure: Can Learning From D Tasks Generalize to DT Tasks?](https://mlsanity.com/paper/2502.08991/) (2025)
- [Leveraging Code to Improve In-context Learning for Semantic Parsing](https://mlsanity.com/paper/2311.09519/) (2023)
- [Compositional API Recommendation for Library-Oriented Code Generation](https://mlsanity.com/paper/2402.19431/) (2024)
- [Move-in-2D: 2D-Conditioned Human Motion Generation](https://mlsanity.com/paper/2412.13185/) (2024)
- [Revisiting In-Context Learning with Long Context Language Models](https://mlsanity.com/paper/2412.16926/) (2024)
- [MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations](https://mlsanity.com/paper/2310.11634/) (2023)
- [When can transformers compositionally generalize in-context?](https://mlsanity.com/paper/2407.12275/) (2024)
- [Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Models](https://mlsanity.com/paper/2311.09064/) (2023)
- [FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation](https://mlsanity.com/paper/2501.00777/) (2025)
- [The Role of Diversity in In-Context Learning for Large Language Models](https://mlsanity.com/paper/2505.19426/) (2025)
- [Towards Understanding the Relationship between In-context Learning and Compositional Generalization](https://mlsanity.com/paper/2403.11834/) (2024)
- [Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions](https://mlsanity.com/paper/2506.11111/) (2025)
- [Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers](https://mlsanity.com/paper/2601.20796/) (2026)
- [Unraveling Indirect In-Context Learning Using Influence Functions](https://mlsanity.com/paper/2501.01473/) (2025)
- [Towards Compute-Optimal Many-Shot In-Context Learning](https://mlsanity.com/paper/2507.16217/) (2025)

## Abstract (arXiv)

Compositional generalization--understanding unseen combinations of seen primitives--is an essential reasoning capability in human intelligence. The AI community mainly studies this capability by fine-tuning neural networks on lots of training samples, while it is still unclear whether and how in-context learning--the prevailing few-shot paradigm based on large language models--exhibits compositional generalization. In this paper, we present CoFe, a test suite to investigate in-context compositional generalization. We find that the compositional generalization performance can be easily affected by the selection of in-context examples, thus raising the research question what the key factors are to make good in-context examples for compositional generalization. We study three potential factors: similarity, diversity and complexity. Our systematic experiments indicate that in-context examples should be structurally similar to the test case, diverse from each other, and individually simple. Furthermore, two strong limitations are observed: in-context compositional generalization on fictional words is much weaker than that on commonly used ones; it is still critical that the in-context examples should cover required linguistic structures, even though the backbone model has been pre-trained on large corpus. We hope our analysis would facilitate the understanding and utilization of in-context learning paradigm.