mlsanityData to 7 Oct 2026

Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations

arXiv 2205.12685 ↗First posted 25 May 2022cs.CL cs.AI cs.LGcited by 116

Not on the map. Why?

Abstract (arXiv)

Despite recent explosion of interests in in-context learning, the underlying mechanism and the precise impact of the quality of demonstrations remain elusive. Intuitively, ground-truth labels should have as much impact in in-context learning (ICL) as supervised learning, but recent work reported that the input-label correspondence is significantly less important than previously thought. Intrigued by this counter-intuitive observation, we re-examine the importance of ground-truth labels in in-context learning. With the introduction of two novel metrics, namely Label-Correctness Sensitivity and Ground-truth Label Effect Ratio (GLER), we were able to conduct quantifiable analysis on the impact of ground-truth label demonstrations. Through extensive analyses, we find that the correct input-label mappings can have varying impacts on the downstream in-context learning performances, depending on the experimental configuration. Through additional studies, we identify key components, such as the verbosity of prompt templates and the language model size, as the controlling factor to achieve more noise-resilient ICL.

Key facts. Cited by 116 papers in the map (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

30 of 43 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
18 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  3. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  4. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  6. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  7. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  8. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  9. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  10. 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
  11. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  12. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  13. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  14. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  15. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  16. 2020ETHOS: an Online Hate Speech Detection Datasetcited by 66
  17. 2020Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQscited by 48
  18. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 41; 2024: 44; 2025: 12; 2026: 8.

116 papers.

Sort:

By area:

  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  3. 2022A Survey on In-context Learningcited by 717
  4. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  5. 2023Function Vectors in Large Language Modelscited by 292
  6. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  7. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  8. 2024Many-Shot In-Context Learningcited by 203
  9. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  10. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  11. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  12. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
38 more
  1. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  2. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  3. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  4. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  5. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  6. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  7. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasonerscited by 63
  8. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  9. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 56
  10. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  11. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  12. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  13. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  14. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  15. 2023Small Models are Valuable Plug-ins for Large Language Modelscited by 42
  16. 2023Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selectioncited by 36
  17. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  18. 2023Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learningcited by 31
  19. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 31
  20. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 29
  21. 2022Task Ambiguity in Humans and Language Modelscited by 27
  22. 2023Understanding In-Context Learning from Repetitionscited by 27
  23. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 26
  24. 2023Generative Calibration for In-context Learningcited by 20
  25. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Promptscited by 19
  26. 2022Prompt-Augmented Linear Probing: Scaling beyond the Limit of Few-shot In-Context Learnerscited by 18
  27. 2023Efficient Prompting via Dynamic In-Context Learningcited by 18
  28. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  29. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 16
  30. 2023Overview of the PromptCBLUE Shared Task in CHIP2023cited by 14
  31. 2023In-Context Exemplars as Clues to Retrieving from Large Associative Memorycited by 13
  32. 2023More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineeringcited by 13
  33. 2024The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysiscited by 13
  34. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  35. 2022Robustness of Demonstration-based Learning Under Limited Data Scenariocited by 11
  36. 2023Investigating the Learning Behaviour of In-context Learning: A Comparison with Supervised Learningcited by 11
  37. 2023Ambiguity-Aware In-Context Learning with Large Language Modelscited by 11
  38. 2023Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editingcited by 11

50 most cited of 116 All 116 (JSON)