mlsanityData to 7 Oct 2026

Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning

arXiv 2305.17256 ↗First posted 26 May 2023cs.CL cs.AI cs.LGcited by 69

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have recently shown great potential for in-context learning, where LLMs learn a new task simply by conditioning on a few input-label pairs (prompts). Despite their potential, our understanding of the factors influencing end-task performance and the robustness of in-context learning remains limited. This paper aims to bridge this knowledge gap by investigating the reliance of LLMs on shortcuts or spurious correlations within prompts. Through comprehensive experiments on classification and extraction tasks, we reveal that LLMs are "lazy learners" that tend to exploit shortcuts in prompts for downstream tasks. Additionally, we uncover a surprising finding that larger models are more likely to utilize shortcuts in prompts during inference. Our findings provide a new perspective on evaluating robustness in in-context learning and pose new challenges for detecting and mitigating the use of shortcuts in prompts.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

31 of 42 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  10. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  12. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
19 more
  1. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  2. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  3. 2021MetaICL: Learning to Learn In Contextcited by 450
  4. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  5. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  6. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  7. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  8. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  9. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  10. 2021Meta-learning via Language Model In-context Tuningcited by 163
  11. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  12. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 147
  13. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  14. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  15. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  16. 2021Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Modelscited by 76
  17. 2020Mitigating Gender Bias in Captioning Systemscited by 50
  18. 2021Why Machine Reading Comprehension Models Learn Shortcuts?cited by 40
  19. 2022Prompt-Augmented Linear Probing: Scaling beyond the Limit of Few-shot In-Context Learnerscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 27; 2025: 17; 2026: 16.

69 papers.

Sort:

By area:

  1. 2023Explore Spurious Correlations at the Concept Level in Language Models for Text Classificationcited by 48
  2. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  3. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  4. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31
  5. 2024Do Vision-Language Models Really Understand Visual Language?cited by 29
  6. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 25
  7. 2024Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learningcited by 23
  8. 2025All Patches Matter, More Patches Better: Enhance AI-Generated Image Detection via Panoptic Patch Learningcited by 23
  9. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21
  10. 2024SLANG: New Concept Comprehension of Large Language Modelscited by 20
  11. 2024LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational Historycited by 19
  12. 2024Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theorycited by 19
38 more
  1. 2023Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypotscited by 16
  2. 2024Can Graph Descriptive Order Affect Solving Graph Problems with LLMs?cited by 16
  3. 2024When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated Explanationscited by 16
  4. 2025AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Modelscited by 16
  5. 2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 14
  6. 2024Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoringcited by 13
  7. 2023Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?cited by 12
  8. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  9. 2024Does learning the right latent variables necessarily improve in-context learning?cited by 12
  10. 2024Creating a Lens of Chinese Culture: A Multimodal Dataset for Chinese Pun Rebus Art Understandingcited by 11
  11. 2024Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language Modelscited by 10
  12. 2024An LLM Feature-based Framework for Dialogue Constructiveness Assessmentcited by 9
  13. 2024In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Modelscited by 8
  14. 2024Shortcut Learning in In-Context Learning: A Surveycited by 8
  15. 2024Rectifying Demonstration Shortcut in In-Context Learningcited by 7
  16. 2024Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templatescited by 7
  17. 2023Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Modelscited by 6
  18. 2023Comparable Demonstrations are Important in In-Context Learning: A Novel Perspective on Demonstration Selectioncited by 5
  19. 2024Leveraging Large Language Models for Enhanced Process Model Comprehensioncited by 5
  20. 2025FaithUn: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledgecited by 5
  21. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  22. 2025Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Trainingcited by 5
  23. 2026TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraintscited by 5
  24. 2023Reinforcement Learning Fine-tuning of Language Models is Biased Towards More Extractable Featurescited by 3
  25. 2023Positional Information Matters for Invariant In-Context Learning: A Case Study of Simple Function Classescited by 3
  26. 2025Simulating Human-Like Learning Dynamics with LLM-Empowered Agentscited by 3
  27. 2025Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Modelscited by 3
  28. 2026The Trap of Trajectory: Towards Understanding and Mitigating Spurious Correlations in Agentic Memorycited by 3
  29. 2024Mitigate Position Bias in Large Language Models via Scaling a Single Dimensioncited by 2
  30. 2024What Makes Language Models Good-enough?cited by 2
  31. 2024In-context Learning in Presence of Spurious Correlationscited by 2
  32. 2024Conceptual In-Context Learning and Chain of Concepts: Solving Complex Conceptual Problems Using Large Language Modelscited by 2
  33. 2025Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Modelscited by 2
  34. 2025Exploring Explanations Improves the Robustness of In-Context Learningcited by 2
  35. 2025Detecting and Pruning Prominent but Detrimental Neurons in Large Language Modelscited by 2
  36. 2025AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programmingcited by 2
  37. 2026CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matchingcited by 2
  38. 2023TMID: A Comprehensive Real-world Dataset for Trademark Infringement Detection in E-Commercecited by 1

50 most cited of 69 All 69 (JSON)