mlsanityData to 7 Oct 2026

Compositional Exemplars for In-context Learning

arXiv 2302.05698 ↗First posted 11 Feb 2023cs.CL cs.AI cs.LGcited by 143

Not on the map. Why?

Abstract (arXiv)

Large pretrained language models (LMs) have shown impressive In-Context Learning (ICL) ability, where the model learns to do an unseen task via a prompt consisting of input-output examples as the demonstration, without any parameter updates. The performance of ICL is highly dominated by the quality of the selected in-context examples. However, previous selection methods are mostly based on simple heuristics, leading to sub-optimal performance. In this work, we formulate in-context example selection as a subset selection problem. We propose CEIL (Compositional Exemplars for In-context Learning), which is instantiated by Determinantal Point Processes (DPPs) to model the interaction between the given input and in-context examples, and optimized through a carefully-designed contrastive learning objective to obtain preference from LMs. We validate CEIL on 12 classification and generation datasets from 7 distinct NLP tasks, including sentiment analysis, paraphrase detection, natural language inference, commonsense reasoning, open-domain question answering, code generation, and semantic parsing. Extensive experiments demonstrate not only the state-of-the-art performance but also the transferability and compositionality of CEIL, shedding new light on effective and efficient in-context learning. Our code is released at https://github.com/HKUNLP/icl-ceil.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). Builds on 40 of its 75 references that are in the map. First posted on arXiv 11 Feb 2023.

Builds on

40 of 75 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
28 more
  1. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  9. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  10. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  11. 2020Extractive Summarization as Text Matchingcited by 189
  12. 2016Movie Descriptioncited by 180
  13. 2022In-context Examples Selection for Machine Translationcited by 172
  14. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  15. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  16. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  17. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  18. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  19. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  20. 2022Natural Language to Code Translation with Executioncited by 122
  21. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 97
  22. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  23. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  24. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  25. 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
  26. 2017Faster Greedy MAP Inference for Determinantal Point Processescited by 14
  27. 2021Question Decomposition with Dependency Graphscited by 7
  28. 2017Learning Detection with Diverse Proposalscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 43; 2025: 51; 2026: 19.

143 papers.

Sort:

By area:

  1. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  2. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  3. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  4. 2023Finding Support Examples for In-Context Learningcited by 104
  5. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  6. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  7. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 74
  8. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  9. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 71
  10. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  11. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  12. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
38 more
  1. 2023Learning to Retrieve In-Context Examples for Large Language Modelscited by 60
  2. 2023Coverage-based Example Selection for In-Context Learningcited by 58
  3. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  4. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  5. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  6. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 47
  7. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  8. 2023Small Models are Valuable Plug-ins for Large Language Modelscited by 42
  9. 2023Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergencecited by 42
  10. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 40
  11. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 39
  12. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 39
  13. 2023MoT: Memory-of-Thought Enables ChatGPT to Self-Improvecited by 37
  14. 2024Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplarscited by 37
  15. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  16. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  17. 2024Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimizationcited by 28
  18. 2023Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Modelscited by 19
  19. 2024Assistive Large Language Model Agents for Socially-Aware Negotiation Dialoguescited by 18
  20. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 15
  21. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  22. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  23. 2024Misconfidence-based Demonstration Selection for LLM In-Context Learningcited by 14
  24. 2023In-Context Exemplars as Clues to Retrieving from Large Associative Memorycited by 13
  25. 2024Learning to Retrieve Iteratively for In-Context Learningcited by 13
  26. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  27. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  28. 2023Generating Data for Symbolic Language with Large Language Modelscited by 12
  29. 2024Visual Prompt Selection for In-Context Learning Segmentationcited by 12
  30. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  31. 2024How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?cited by 11
  32. 2023Leveraging Code to Improve In-context Learning for Semantic Parsingcited by 10
  33. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  34. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  35. 2023SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQAcited by 9
  36. 2023Improving In-Context Learning in Diffusion Models with Visual Context-Modulated Promptscited by 9
  37. 2024NICE: To Optimize In-Context Examples or Not?cited by 9
  38. 2024Learning to Reason via Program Generation, Emulation, and Searchcited by 9

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting
  2. 2024In-Context Language Learning: Architectures and Algorithms
  3. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  4. 2023Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models
  5. 2025Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
  6. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
  7. 2025Order Matters: Rethinking Prompt Construction in In-Context Learning
  8. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning