# Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations

arXiv: https://arxiv.org/abs/2205.12685 · HTML: https://mlsanity.com/paper/2205.12685/

## Key facts

- Cited by 116 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts](https://mlsanity.com/paper/1307.5336/) (2013)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model](https://mlsanity.com/paper/2204.13509/) (2022)
- [What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers](https://mlsanity.com/paper/2109.04650/) (2021)
- [ETHOS: an Online Hate Speech Detection Dataset](https://mlsanity.com/paper/2006.08328/) (2020)
- [Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQs](https://mlsanity.com/paper/2008.13546/) (2020)
- [Investigating Societal Biases in a Poetry Composition System](https://mlsanity.com/paper/2011.02686/) (2020)

## Built on by (most cited first, 30 of 116)

- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://mlsanity.com/paper/2212.10001/) (2022)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering](https://mlsanity.com/paper/2212.10375/) (2022)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning](https://mlsanity.com/paper/2305.09731/) (2023)
- [When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards](https://mlsanity.com/paper/2402.01781/) (2024)
- [Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango](https://mlsanity.com/paper/2209.07686/) (2022)
- [Structured Prompting: Scaling In-Context Learning to 1,000 Examples](https://mlsanity.com/paper/2212.06713/) (2022)
- [Overthinking the Truth: Understanding how Language Models Process False Demonstrations](https://mlsanity.com/paper/2307.09476/) (2023)
- [Boosting Language Models Reasoning with Chain-of-Knowledge Prompting](https://mlsanity.com/paper/2306.06427/) (2023)
- [In-Context Learning with Iterative Demonstration Selection](https://mlsanity.com/paper/2310.09881/) (2023)
- [On the Relation between Sensitivity and Accuracy in In-context Learning](https://mlsanity.com/paper/2209.07661/) (2022)
- [Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners](https://mlsanity.com/paper/2305.14825/) (2023)
- [PersonalLLM: Tailoring LLMs to Individual Preferences](https://mlsanity.com/paper/2409.20296/) (2024)
- [Data Curation Alone Can Stabilize In-context Learning](https://mlsanity.com/paper/2212.10378/) (2022)
- [What Makes Multimodal In-Context Learning Work?](https://mlsanity.com/paper/2404.15736/) (2024)
- [Active Learning Principles for In-Context Learning with Large Language Models](https://mlsanity.com/paper/2305.14264/) (2023)
- [In-Context Learning Learns Label Relationships but Is Not Conventional Learning](https://mlsanity.com/paper/2307.12375/) (2023)
- [Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression](https://mlsanity.com/paper/2310.17086/) (2023)
- [Exploring Effective Factors for Improving Visual In-Context Learning](https://mlsanity.com/paper/2304.04748/) (2023)
- [Small Models are Valuable Plug-ins for Large Language Models](https://mlsanity.com/paper/2305.08848/) (2023)
- [Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection](https://mlsanity.com/paper/2310.20046/) (2023)
- [The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis](https://mlsanity.com/paper/2311.00237/) (2023)
- [Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?](https://mlsanity.com/paper/2311.18021/) (2023)

## Abstract (arXiv)

Despite recent explosion of interests in in-context learning, the underlying mechanism and the precise impact of the quality of demonstrations remain elusive. Intuitively, ground-truth labels should have as much impact in in-context learning (ICL) as supervised learning, but recent work reported that the input-label correspondence is significantly less important than previously thought. Intrigued by this counter-intuitive observation, we re-examine the importance of ground-truth labels in in-context learning. With the introduction of two novel metrics, namely Label-Correctness Sensitivity and Ground-truth Label Effect Ratio (GLER), we were able to conduct quantifiable analysis on the impact of ground-truth label demonstrations. Through extensive analyses, we find that the correct input-label mappings can have varying impacts on the downstream in-context learning performances, depending on the experimental configuration. Through additional studies, we identify key components, such as the verbosity of prompt templates and the language model size, as the controlling factor to achieve more noise-resilient ICL.