mlsanityData to 7 Oct 2026

Rectifying Demonstration Shortcut in In-Context Learning

arXiv 2403.09488 ↗First posted 14 Mar 2024cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are able to solve various tasks with only a few demonstrations utilizing their in-context learning (ICL) abilities. However, LLMs often rely on their pre-trained semantic priors of demonstrations rather than on the input-label relationships to proceed with ICL prediction. In this work, we term this phenomenon as the 'Demonstration Shortcut'. While previous works have primarily focused on improving ICL prediction results for predefined tasks, we aim to rectify the Demonstration Shortcut, thereby enabling the LLM to effectively learn new input-label relationships from demonstrations. To achieve this, we introduce In-Context Calibration, a demonstration-aware calibration method. We evaluate the effectiveness of the proposed method in two settings: (1) the Original ICL Task using the standard label space and (2) the Task Learning setting, where the label space is replaced with semantically unrelated tokens. In both settings, In-Context Calibration demonstrates substantial improvements, with results generalized across three LLM families (OPT, GPT, and Llama2) under various configurations.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

25 of 44 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  12. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
13 more
  1. 2023Larger language models do in-context learning differentlycited by 386
  2. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  3. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  4. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  5. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  6. 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
  7. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  8. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  9. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  10. 2023Mitigating Label Biases for In-context Learningcited by 88
  11. 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
  12. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 47
  13. 2023Generative Calibration for In-context Learningcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2024Shortcut Learning in In-Context Learning: A Surveycited by 8
  2. 2026Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPOcited by 3
  3. 2025Exploring Explanations Improves the Robustness of In-Context Learningcited by 2
  4. 2025Context Tuning for In-Context Optimizationcited by 2
  5. 2026PLR: Plackett-Luce for Reordering In-Context Learning Examplescited by 1
  6. 2026Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanationscited by 0
  7. 2026When Context Misleads: In-context Learning with Jurisdiction in Large Language Modelscited by 0