mlsanityData to 7 Oct 2026

Selective Annotation Makes Language Models Better Few-Shot Learners

arXiv 2209.01975 ↗First posted 5 Sep 2022cs.CLcited by 242

Not on the map. Why?

Cited by 25 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Many recent approaches to natural language tasks are built on the remarkable abilities of large language models. Large language models can perform in-context learning, where they learn a new task from a few task demonstrations, without any parameter updates. This work examines the implications of in-context learning for the creation of datasets for new natural language tasks. Departing from recent in-context learning methods, we formulate an annotation-efficient, two-step framework: selective annotation that chooses a pool of examples to annotate from unlabeled data in advance, followed by prompt retrieval that retrieves task examples from the annotated pool at test time. Based on this framework, we propose an unsupervised, graph-based selective annotation method, voke-k, to select diverse, representative examples to annotate. Extensive experiments on 10 datasets (covering classification, commonsense reasoning, dialogue, and text/code generation) demonstrate that our selective annotation method improves the task performance by a large margin. On average, vote-k achieves a 12.9%/11.4% relative gain under an annotation budget of 18/100, as compared to randomly selecting examples to annotate. Compared to state-of-the-art supervised finetuning approaches, it yields similar performance with 10-100x less annotation cost across 10 tasks. We further analyze the effectiveness of our framework in various scenarios: language models with varying sizes, alternative selective annotation methods, and cases where there is a test data domain shift. We hope that our studies will serve as a basis for data annotations as large language models are increasingly applied to new tasks. Our code is available at https://github.com/HKUNLP/icl-selective-annotation.

Key facts. Cited by 242 papers in the map (data through 7 Oct 2026). Builds on 40 of its 59 references that are in the map. First posted on arXiv 5 Sep 2022.

Builds on

40 of 59 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
28 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  6. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  12. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  13. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  14. 2021MetaICL: Learning to Learn In Contextcited by 450
  15. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  16. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  17. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  18. 2017Cost-Effective Active Learning for Deep Image Classificationcited by 165
  19. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  20. 2021Meta-learning via Language Model In-context Tuningcited by 163
  21. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  22. 2020Semantic Evaluation for Text-to-SQL with Distilled Test Suitescited by 123
  23. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  24. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  25. 2019Low-resource Deep Entity Resolution with Transfer and Active Learningcited by 37
  26. 2022Active Learning Helps Pretrained Models Learn the Intended Taskcited by 33
  27. 2022Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Trackingcited by 21
  28. 2022Active Learning Over Multiple Domains in Natural Language Taskscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 88; 2024: 79; 2025: 43; 2026: 18.

242 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  3. 2022A Survey on In-context Learningcited by 717
  4. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  5. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  6. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  7. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  8. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  9. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  10. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  11. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  12. 2022Binding Language Models in Symbolic Languagescited by 224
38 more
  1. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  2. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  3. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  4. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  5. 2023Efficient Large Language Models: A Surveycited by 183
  6. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  7. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  8. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  9. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  10. 2023Compositional Exemplars for In-context Learningcited by 143
  11. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  12. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  13. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  14. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  15. 2023Large Language Models as Analogical Reasonerscited by 107
  16. 2023Finding Support Examples for In-Context Learningcited by 104
  17. 2023Text Classification via Large Language Modelscited by 104
  18. 2023WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferencescited by 100
  19. 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environmentscited by 93
  20. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 84
  21. 2024RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agentscited by 80
  22. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  23. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  24. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  25. 2023Exploring Chain-of-Thought Style Prompting for Text-to-SQLcited by 69
  26. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  27. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  28. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  29. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  30. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  31. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasonerscited by 63
  32. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
  33. 2023In-context Example Selection with Influencescited by 60
  34. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  35. 2023Coverage-based Example Selection for In-Context Learningcited by 58
  36. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 57
  37. 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 56
  38. 2023Are LLMs All You Need for Task-Oriented Dialogue?cited by 52

50 most cited of 242 All 242 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Learning to Retrieve In-Context Examples for Large Language Models
  2. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
  3. 2025Optimising Language Models for Downstream Tasks: A Post-Training Perspective
  4. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  5. 2024Many-Shot In-Context Learning
  6. 2021Self-training with Few-shot Rationalization: Teacher Explanations Aid Student in Few-shot NLU
  7. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation
  8. 2024In-Context Language Learning: Architectures and Algorithms