# Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps

arXiv: https://arxiv.org/abs/2307.05052 · HTML: https://mlsanity.com/paper/2307.05052/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 11 Jul 2023.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [Visual News: Benchmark and Challenges in News Image Captioning](https://mlsanity.com/paper/2010.03743/) (2020)
- [Complementary Explanations for Effective In-Context Learning](https://mlsanity.com/paper/2211.13892/) (2022)
- [DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents](https://mlsanity.com/paper/2306.06306/) (2023)

## Built on by (most cited first, 30 of 35)

- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries](https://mlsanity.com/paper/2310.13132/) (2023)
- [From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning](https://mlsanity.com/paper/2310.00492/) (2023)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Neuron-Level Knowledge Attribution in Large Language Models](https://mlsanity.com/paper/2312.12141/) (2023)
- [Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey](https://mlsanity.com/paper/2407.21794/) (2024)
- [XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models](https://mlsanity.com/paper/2407.15248/) (2024)
- [The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Models](https://mlsanity.com/paper/2502.08009/) (2025)
- [Explainable Chain-of-Thought Reasoning: An Empirical Analysis on State-Aware Reasoning Dynamics](https://mlsanity.com/paper/2509.00190/) (2025)
- [LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing](https://mlsanity.com/paper/2505.16491/) (2025)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Memorization in In-Context Learning](https://mlsanity.com/paper/2408.11546/) (2024)
- [PromptExp: Multi-granularity Prompt Explanation of Large Language Models](https://mlsanity.com/paper/2410.13073/) (2024)
- [Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation](https://mlsanity.com/paper/2412.03944/) (2024)
- [Towards detecting unanticipated bias in Large Language Models](https://mlsanity.com/paper/2404.02650/) (2024)
- [Evaluating the Reliability of Self-Explanations in Large Language Models](https://mlsanity.com/paper/2407.14487/) (2024)
- [Multimodal Contrastive In-Context Learning](https://mlsanity.com/paper/2408.12959/) (2024)
- [Reassessing the Role of Chain-of-Thought in Sentiment Analysis: Insights and Limitations](https://mlsanity.com/paper/2501.08641/) (2025)
- [Towards Transparent AI: A Survey on Explainable Language Models](https://mlsanity.com/paper/2509.21631/) (2025)
- [NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors](https://mlsanity.com/paper/2602.22144/) (2026)
- [From Actions to Understanding: Conformal Interpretability of Temporal Concepts in LLM Agents](https://mlsanity.com/paper/2604.19775/) (2026)
- [Which Words Matter Most in Zero-Shot Prompts?](https://mlsanity.com/paper/2502.03418/) (2025)
- [Large Language Models Struggle to Describe the Haystack without Human Help: Human-in-the-loop Evaluation of Topic Models](https://mlsanity.com/paper/2502.14748/) (2025)
- [Can Large Language Models Simulate Human Responses? A Case Study of Stated Preference Experiments in the Context of Heating-related Choices](https://mlsanity.com/paper/2503.10652/) (2025)
- [Understanding the Dynamics of Demonstration Conflict in In-Context Learning](https://mlsanity.com/paper/2603.04464/) (2026)
- [Large language models reorganize representational geometry during in-context learning](https://mlsanity.com/paper/2605.28854/) (2026)
- [Rethinking Distance Metrics for Counterfactual Explainability](https://mlsanity.com/paper/2410.14522/) (2024)
- [NEAT: Concept driven Neuron Attribution in LLMs](https://mlsanity.com/paper/2508.15875/) (2025)

## Abstract (arXiv)

We investigate the role of various demonstration components in the in-context learning (ICL) performance of large language models (LLMs). Specifically, we explore the impacts of ground-truth labels, input distribution, and complementary explanations, particularly when these are altered or perturbed. We build on previous work, which offers mixed findings on how these elements influence ICL. To probe these questions, we employ explainable NLP (XNLP) methods and utilize saliency maps of contrastive demonstrations for both qualitative and quantitative analysis. Our findings reveal that flipping ground-truth labels significantly affects the saliency, though it's more noticeable in larger LLMs. Our analysis of the input distribution at a granular level reveals that changing sentiment-indicative terms in a sentiment analysis task to neutral ones does not have as substantial an impact as altering ground-truth labels. Finally, we find that the effectiveness of complementary explanations in boosting ICL performance is task-dependent, with limited benefits seen in sentiment analysis tasks compared to symbolic reasoning tasks. These insights are critical for understanding the functionality of LLMs and guiding the development of effective demonstrations, which is increasingly relevant in light of the growing use of LLMs in applications such as ChatGPT. Our research code is publicly available at https://github.com/paihengxu/XICL.