mlsanityData to 7 Oct 2026

Shortcut Learning in In-Context Learning: A Survey

arXiv 2411.02018 ↗First posted 4 Nov 2024cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Shortcut learning refers to the phenomenon where models employ simple, non-robust decision rules in practical tasks, which hinders their generalization and robustness. With the rapid development of large language models (LLMs) in recent years, an increasing number of studies have shown the impact of shortcut learning on LLMs. This paper provides a novel perspective to review relevant research on shortcut learning in In-Context Learning (ICL). It conducts a detailed exploration of the types of shortcuts in ICL tasks, their causes, available benchmarks, and strategies for mitigating shortcuts. Based on corresponding observations, it summarizes the unresolved issues in existing research and attempts to outline the future research landscape of shortcut learning.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 64 of its 88 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

64 of 88 references.

References · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  10. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
38 more
  1. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  2. 2022A Survey on In-context Learningcited by 717
  3. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  4. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  5. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  6. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  7. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
  8. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  9. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  10. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  11. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  12. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  13. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  14. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  15. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  16. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  17. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  18. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 147
  19. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  20. 2023Mitigating Label Biases for In-context Learningcited by 88
  21. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 83
  22. 2020Learning from others' mistakes: Avoiding dataset biases without modeling themcited by 83
  23. 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
  24. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
  25. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  26. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  27. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  28. 2023Explore Spurious Correlations at the Concept Level in Language Models for Text Classificationcited by 48
  29. 2020What do Models Learn from Question Answering Datasets?cited by 45
  30. 2021Why Machine Reading Comprehension Models Learn Shortcuts?cited by 40
  31. 2020Towards Robustifying NLI Models Against Lexical Dataset Biasescited by 34
  32. 2024Examining the robustness of LLM evaluation to the distributional assumptions of benchmarkscited by 31
  33. 2022Should We Rely on Entity Mentions for Relation Extraction? Debiasing Relation Extraction with Counterfactual Analysiscited by 30
  34. 2024UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulationcited by 30
  35. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 29
  36. 2021MATE-KD: Masked Adversarial TExt, a Companion to Knowledge Distillationcited by 27
  37. 2023Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?cited by 26
  38. 2023Mitigating Biases for Instruction-following Language Models via Bias Neurons Eliminationcited by 23

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 25
  2. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  3. 2026Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learningcited by 4
  4. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  5. 2026Deciphering Shortcut Learning from an Evolutionary Game Theory Perspectivecited by 0
  6. 2026Toward Robust In-Context Learning: Leveraging Out-of-distribution Proxies for Target Inaccessible Demonstration Retrievalcited by 0
  7. 2026Diagnosing Task Insensitivity in Language Agentscited by 0
  8. 2026Beyond the Shadows of Plato's Cave: Evaluating False Memory in Autonomous Agents via Counterfactual Reasoningcited by 0