Data to 7 Oct 2026

On the Relation between Sensitivity and Accuracy in In-context Learning

arXiv 2209.07661 ↗First posted 16 Sep 2022cs.CL cs.AI cs.LGcited by 71

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) suffers from oversensitivity to the prompt, making it unreliable in real-world scenarios. We study the sensitivity of ICL with respect to multiple perturbation types. First, we find that label bias obscures the true sensitivity, and therefore prior work may have significantly underestimated ICL sensitivity. Second, we observe a strong negative correlation between ICL sensitivity and accuracy: predictions sensitive to perturbations are less likely to be correct. Motivated by these findings, we propose textscSenSel, a few-shot selective prediction method that abstains from sensitive predictions. Experiments on ten classification datasets show that textscSenSel consistently outperforms two commonly used confidence-based and entropy-based baselines on abstention decisions.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 16 Sep 2022.

Builds on

27 of 45 references.

References · 27

2020Language Models are Few-Shot Learnerscited by 32,199
2017On Calibration of Modern Neural Networkscited by 4,860
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
2015Character-level Convolutional Networks for Text Classificationcited by 2,293
2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915

15 more

2017Selective Classification for Deep Neural Networkscited by 799
2021MetaICL: Learning to Learn In Contextcited by 450
2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
2021True Few-Shot Learning with Language Modelscited by 343
2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
2020Selective Question Answering under Domain Shiftcited by 228
2021Meta-learning via Language Model In-context Tuningcited by 163
2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
2015Posterior calibration and exploratory analysis for natural language processing modelscited by 75
2022Prompt Consistency for Zero-Shot Task Generalizationcited by 66
2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
2021Sensitivity as a Complexity Measure for Sequence Classification Taskscited by 28
2015Text Understanding from Scratchcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 19; 2024: 25; 2025: 14; 2026: 6.

71 papers.

Sort:

By area:

2022A Survey on In-context Learningcited by 717
2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
2023Unified Demonstration Retriever for In-Context Learningcited by 141
2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
2023Finding Support Examples for In-Context Learningcited by 104
2023In-context Example Selection with Influencescited by 60
2022Data Curation Alone Can Stabilize In-context Learningcited by 58
2023Adversarial Demonstration Attacks on Large Language Modelscited by 58
2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49
2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46

38 more

2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
2024Social Bias Evaluation for Large Language Models Requires Prompt Variationscited by 39
2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 36
2023Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selectioncited by 36
2023How are Prompts Different in Terms of Sensitivity?cited by 34
2024PromptWizard: Task-Aware Prompt Optimization Frameworkcited by 32
2022Large Language Models can Implement Policy Iterationcited by 31
2024Calibrating Long-form Generations from Large Language Modelscited by 31
2024Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimizationcited by 28
2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 27
2024Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuningcited by 26
2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 24
2024Style-Specific Neurons for Steering LLMs in Text Style Transfercited by 24
2022LMentry: A Language Model Benchmark of Elementary Language Taskscited by 21
2025Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learningcited by 18
2023Editable User Profiles for Controllable Text Recommendationcited by 16
2023Cost-Effective In-Context Learning for Entity Resolution: A Design Space Explorationcited by 15
2024Prompt Stability Scoring for Text Annotation with Large Language Modelscited by 13
2023InstructEval: Systematic Evaluation of Instruction Selection Methodscited by 12
2024XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inferencecited by 12
2024In-Context Learning Demonstration Selection via Influence Analysiscited by 11
2024Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learningcited by 10
2023Estimating Large Language Model Capabilities without Labeled Test Datacited by 9
2024Se2: Sequential Example Selection for In-Context Learningcited by 8
2024The Mystery of the Pathological Path-star Task for Language Modelscited by 8
2025Rethinking Invariance in In-context Learningcited by 8
2025Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attentioncited by 8
2024When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Modelscited by 6
2025The Role of Diversity in In-Context Learning for Large Language Modelscited by 6
2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 6
2023Flatness-Aware Prompt Selection Improves Accuracy and Sample Efficiencycited by 5
2023ICL Markup: Structuring In-Context Learning using Soft-Token Tagscited by 5
2024Unveiling the Misuse Potential of Base Large Language Models via In-Context Learningcited by 5
2024Mixtures of In-Context Learnerscited by 5
2025AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Modelscited by 5
2025Learning to Select In-Context Demonstration Preferred by Large Language Modelcited by 5
2024A Study on Training and Developing Large Language Models for Behavior Tree Generationcited by 4
2024Knowledge Editing on Black-box Large Language Modelscited by 4

50 most cited of 71 All 71 (JSON)