# On the Compositional Generalization Gap of In-Context Learning

arXiv: https://arxiv.org/abs/2211.08473 · HTML: https://mlsanity.com/paper/2211.08473/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?](https://mlsanity.com/paper/2204.05832/) (2022)
- [OpenPrompt: An Open-source Framework for Prompt-learning](https://mlsanity.com/paper/2111.01998/) (2021)
- [Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?](https://mlsanity.com/paper/2010.12725/) (2020)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners](https://mlsanity.com/paper/2205.10747/) (2022)
- [Compositional generalization through meta sequence-to-sequence learning](https://mlsanity.com/paper/1906.05381/) (2019)
- [Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures](https://mlsanity.com/paper/2007.08970/) (2020)
- [Span-based Semantic Parsing for Compositional Generalization](https://mlsanity.com/paper/2009.06040/) (2020)
- [Compositional generalization in a deep seq2seq model by separating syntax and semantics](https://mlsanity.com/paper/1904.09708/) (2019)
- [Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations](https://mlsanity.com/paper/2104.07478/) (2021)
- [Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing](https://mlsanity.com/paper/2205.12253/) (2022)
- [Compositional Generalization for Primitive Substitutions](https://mlsanity.com/paper/1910.02612/) (2019)
- [AdaPrompt: Adaptive Model Training for Prompt-based NLP](https://mlsanity.com/paper/2202.04824/) (2022)
- [Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks](https://mlsanity.com/paper/2109.15256/) (2021)
- [*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task](https://mlsanity.com/paper/2012.08266/) (2020)
- [Neural Compositional Denotational Semantics for Question Answering](https://mlsanity.com/paper/1808.09942/) (2018)

## Built on by (most cited first, 24 of 24)

- [Diverse Demonstrations Improve In-context Compositional Generalization](https://mlsanity.com/paper/2212.06800/) (2022)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples](https://mlsanity.com/paper/2305.15269/) (2023)
- [How Do In-Context Examples Affect Compositional Generalization?](https://mlsanity.com/paper/2305.04835/) (2023)
- [Instruct Me More! Random Prompting for Visual In-Context Learning](https://mlsanity.com/paper/2311.03648/) (2023)
- [Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks](https://mlsanity.com/paper/2311.12997/) (2023)
- [Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoning](https://mlsanity.com/paper/2405.06680/) (2024)
- [Attention as a Hypernetwork](https://mlsanity.com/paper/2406.05816/) (2024)
- [The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks](https://mlsanity.com/paper/2310.17514/) (2023)
- [Leveraging Code to Improve In-context Learning for Semantic Parsing](https://mlsanity.com/paper/2311.09519/) (2023)
- [Adapt and Decompose: Efficient Generalization of Text-to-SQL via Domain Adapted Least-To-Most Prompting](https://mlsanity.com/paper/2308.02582/) (2023)
- [When can transformers compositionally generalize in-context?](https://mlsanity.com/paper/2407.12275/) (2024)
- [Towards Understanding the Relationship between In-context Learning and Compositional Generalization](https://mlsanity.com/paper/2403.11834/) (2024)
- [A General Theory for Compositional Generalization](https://mlsanity.com/paper/2405.11743/) (2024)
- [Positional Information Matters for Invariant In-Context Learning: A Case Study of Simple Function Classes](https://mlsanity.com/paper/2311.18194/) (2023)
- [Neuro-symbolic Zero-Shot Code Cloning with Cross-Language Intermediate Representation](https://mlsanity.com/paper/2304.13350/) (2023)
- [Power-up! What Can Generative Models Do for Human Computation Workflows?](https://mlsanity.com/paper/2307.02243/) (2023)
- [DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalization](https://mlsanity.com/paper/2406.04669/) (2024)
- [Compositional Generalization with Grounded Language Models](https://mlsanity.com/paper/2406.04989/) (2024)
- [Learning to Substitute Components for Compositional Generalization](https://mlsanity.com/paper/2502.20834/) (2025)
- [E-InMeMo: Enhanced Prompting for Visual In-Context Learning](https://mlsanity.com/paper/2504.18158/) (2025)
- [PANICL: Mitigating Over-Reliance on Single Prompt in Visual In-Context Learning](https://mlsanity.com/paper/2509.21926/) (2025)
- [In-Context Compositional Learning via Sparse Coding Transformer](https://mlsanity.com/paper/2511.20194/) (2025)
- [Theory on Attention Dynamics for Out-of-Distribution In-Context Learning](https://mlsanity.com/paper/2609.36448/) (2026)

## Abstract (arXiv)

Pretrained large generative language models have shown great performance on many tasks, but exhibit low compositional generalization abilities. Scaling such models has been shown to improve their performance on various NLP tasks even just by conditioning them on a few examples to solve the task without any fine-tuning (also known as in-context learning). In this work, we look at the gap between the in-distribution (ID) and out-of-distribution (OOD) performance of such models in semantic parsing tasks with in-context learning. In the ID settings, the demonstrations are from the same split (test or train) that the model is being evaluated on, and in the OOD settings, they are from the other split. We look at how the relative generalization gap of in-context learning evolves as models are scaled up. We evaluate four model families, OPT, BLOOM, CodeGen and Codex on three semantic parsing datasets, CFQ, SCAN and GeoQuery with different number of exemplars, and observe a trend of decreasing relative generalization gap as models are scaled up.