mlsanityData to 7 Oct 2026

On the Compositional Generalization Gap of In-Context Learning

arXiv 2211.08473 ↗First posted 15 Nov 2022cs.CL cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Pretrained large generative language models have shown great performance on many tasks, but exhibit low compositional generalization abilities. Scaling such models has been shown to improve their performance on various NLP tasks even just by conditioning them on a few examples to solve the task without any fine-tuning (also known as in-context learning). In this work, we look at the gap between the in-distribution (ID) and out-of-distribution (OOD) performance of such models in semantic parsing tasks with in-context learning. In the ID settings, the demonstrations are from the same split (test or train) that the model is being evaluated on, and in the OOD settings, they are from the other split. We look at how the relative generalization gap of in-context learning evolves as models are scaled up. We evaluate four model families, OPT, BLOOM, CodeGen and Codex on three semantic parsing datasets, CFQ, SCAN and GeoQuery with different number of exemplars, and observe a trend of decreasing relative generalization gap as models are scaled up.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

29 of 39 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  11. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  12. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
17 more
  1. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  2. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  3. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  4. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  5. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  6. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  7. 2019Compositional generalization through meta sequence-to-sequence learningcited by 98
  8. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  9. 2020Span-based Semantic Parsing for Compositional Generalizationcited by 77
  10. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  11. 2021Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representationscited by 55
  12. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  13. 2019Compositional Generalization for Primitive Substitutionscited by 48
  14. 2022AdaPrompt: Adaptive Model Training for Prompt-based NLPcited by 30
  15. 2021Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Taskscited by 19
  16. 2020*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Taskcited by 16
  17. 2018Neural Compositional Denotational Semantics for Question Answeringcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 11; 2024: 7; 2025: 4; 2026: 1.

24 papers.

Sort:

By area:

  1. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  2. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  3. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  4. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  5. 2023Instruct Me More! Random Prompting for Visual In-Context Learningcited by 30
  6. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  7. 2024Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoningcited by 15
  8. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
  9. 2024Attention as a Hypernetworkcited by 12
  10. 2023Leveraging Code to Improve In-context Learning for Semantic Parsingcited by 10
  11. 2023Adapt and Decompose: Efficient Generalization of Text-to-SQL via Domain Adapted Least-To-Most Promptingcited by 8
  12. 2024When can transformers compositionally generalize in-context?cited by 7
12 more
  1. 2024Towards Understanding the Relationship between In-context Learning and Compositional Generalizationcited by 5
  2. 2024A General Theory for Compositional Generalizationcited by 4
  3. 2023Positional Information Matters for Invariant In-Context Learning: A Case Study of Simple Function Classescited by 3
  4. 2023Neuro-symbolic Zero-Shot Code Cloning with Cross-Language Intermediate Representationcited by 2
  5. 2023Power-up! What Can Generative Models Do for Human Computation Workflows?cited by 1
  6. 2024DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalizationcited by 1
  7. 2024Compositional Generalization with Grounded Language Modelscited by 1
  8. 2025Learning to Substitute Components for Compositional Generalizationcited by 1
  9. 2025E-InMeMo: Enhanced Prompting for Visual In-Context Learningcited by 1
  10. 2025PANICL: Mitigating Over-Reliance on Single Prompt in Visual In-Context Learningcited by 1
  11. 2025In-Context Compositional Learning via Sparse Coding Transformercited by 0
  12. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0