mlsanityData to 7 Oct 2026

How Do In-Context Examples Affect Compositional Generalization?

arXiv 2305.04835 ↗First posted 8 May 2023cs.CL cs.AIcited by 51

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Compositional generalization--understanding unseen combinations of seen primitives--is an essential reasoning capability in human intelligence. The AI community mainly studies this capability by fine-tuning neural networks on lots of training samples, while it is still unclear whether and how in-context learning--the prevailing few-shot paradigm based on large language models--exhibits compositional generalization. In this paper, we present CoFe, a test suite to investigate in-context compositional generalization. We find that the compositional generalization performance can be easily affected by the selection of in-context examples, thus raising the research question what the key factors are to make good in-context examples for compositional generalization. We study three potential factors: similarity, diversity and complexity. Our systematic experiments indicate that in-context examples should be structurally similar to the test case, diverse from each other, and individually simple. Furthermore, two strong limitations are observed: in-context compositional generalization on fictional words is much weaker than that on commonly used ones; it is still critical that the in-context examples should cover required linguistic structures, even though the backbone model has been pre-trained on large corpus. We hope our analysis would facilitate the understanding and utilization of in-context learning paradigm.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 39 of its 55 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

39 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language Models of Code are Few-Shot Commonsense Learners13 Oct

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
26 more
  1. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  2. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  3. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  4. 2022Active Example Selection for In-Context Learningcited by 195
  5. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  6. 2019Good-Enough Compositional Data Augmentationcited by 158
  7. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  8. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  9. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  10. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  11. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  12. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  13. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  14. 2020Span-based Semantic Parsing for Compositional Generalizationcited by 77
  15. 2020Learning to Recombine and Resample Data for Compositional Generalizationcited by 65
  16. 2021Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representationscited by 55
  17. 2020Compositional Generalization by Learning Analytical Expressionscited by 53
  18. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  19. 2021Learning Algebraic Recombination for Compositional Generalizationcited by 31
  20. 2022Unobserved Local Structures Make Compositional Generalization Hardcited by 29
  21. 2021Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalizationcited by 27
  22. 2022Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Modelscited by 26
  23. 2022On the Compositional Generalization Gap of In-Context Learningcited by 24
  24. 2020Revisiting Iterative Back-Translation from the Perspective of Compositional Generalizationcited by 19
  25. 2021Learning to Generalize Compositionally by Transferring Across Semantic Parsing Taskscited by 6
  26. 2023Does Deep Learning Learn to Abstract? A Systematic Probing Frameworkcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 18; 2025: 18; 2026: 4.

51 papers.

Sort:

By area:

  1. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  2. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  3. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  4. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  5. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  6. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 39
  7. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 31
  8. 2023Instruct Me More! Random Prompting for Visual In-Context Learningcited by 30
  9. 2023Understanding In-Context Learning from Repetitionscited by 27
  10. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 16
  11. 2024Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Modelscited by 12
  12. 2024Attention as a Hypernetworkcited by 12
38 more
  1. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  2. 2024How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?cited by 11
  3. 2024STAND-Guard: A Small Task-Adaptive Content Moderation Modelcited by 11
  4. 2025Task Generalization With AutoRegressive Compositional Structure: Can Learning From D Tasks Generalize to DT Tasks?cited by 11
  5. 2023Leveraging Code to Improve In-context Learning for Semantic Parsingcited by 10
  6. 2024Compositional API Recommendation for Library-Oriented Code Generationcited by 10
  7. 2024Move-in-2D: 2D-Conditioned Human Motion Generationcited by 8
  8. 2023MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretationscited by 7
  9. 2024When can transformers compositionally generalize in-context?cited by 7
  10. 2024Revisiting In-Context Learning with Long Context Language Modelscited by 7
  11. 2023Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Modelscited by 6
  12. 2025FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generationcited by 6
  13. 2025The Role of Diversity in In-Context Learning for Large Language Modelscited by 6
  14. 2024Towards Understanding the Relationship between In-context Learning and Compositional Generalizationcited by 5
  15. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  16. 2026Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformerscited by 5
  17. 2025Unraveling Indirect In-Context Learning Using Influence Functionscited by 3
  18. 2025Towards Compute-Optimal Many-Shot In-Context Learningcited by 3
  19. 2025What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectivescited by 3
  20. 2023Harnessing Dataset Cartography for Improved Compositional Generalization in Transformerscited by 2
  21. 2023Towards Equipping Transformer with the Ability of Systematic Compositionalitycited by 2
  22. 2024Evaluating Structural Generalization in Neural Machine Translationcited by 2
  23. 2025Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysiscited by 2
  24. 2026Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agentscited by 2
  25. 2024DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalizationcited by 1
  26. 2025Learning to Substitute Components for Compositional Generalizationcited by 1
  27. 2025Diverse In-Context Example Selection After Decomposing Programs and Aligned Utterances Improves Semantic Parsingcited by 1
  28. 2025Reinforcing Compositional Retrieval: Retrieving Step-by-Step for Composing Informative Contextscited by 1
  29. 2025E-InMeMo: Enhanced Prompting for Visual In-Context Learningcited by 1
  30. 2025CrossICL: Cross-Task In-Context Learning via Unsupervised Demonstration Transfercited by 1
  31. 2025PANICL: Mitigating Over-Reliance on Single Prompt in Visual In-Context Learningcited by 1
  32. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  33. 2026Learning to Select Visual In-Context Demonstrationscited by 1
  34. 2024Distilling Event Sequence Knowledge From Large Language Modelscited by 0
  35. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learningcited by 0
  36. 2025Investigating the (De)Composition Capabilities of Large Language Models in Natural-to-Formal Language Conversioncited by 0
  37. 2025Can Language Models Compose Skills In-Context?cited by 0
  38. 2025Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspectivecited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models
  2. 2023Can LLM find the green circle? Investigation and Human-guided tool manipulation for compositional generalization
  3. 2025Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Models
  4. 2023Im-Promptu: In-Context Composition from Image Prompts
  5. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
  6. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning
  7. 2025On the generalization of language models from in-context learning and finetuning: a controlled study
  8. 2023Teaching Smaller Language Models To Generalise To Unseen Compositional Questions