mlsanityData to 7 Oct 2026

Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspective

arXiv 2512.12175 ↗First posted 13 Dec 2025cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) perform in-context learning (ICL) with minimal supervised examples, which benefits various natural language processing (NLP) tasks. One of the critical research focus is the selection of prompt demonstrations. Current approaches typically employ retrieval models to select the top-K most semantically similar examples as demonstrations. However, we argue that existing methods are limited since the label consistency is not guaranteed during demonstration selection. Our cognition derives from the Bayesian view of ICL and our rethinking of ICL from the transductive label propagation perspective. We treat ICL as a transductive learning method and incorporate latent concepts from Bayesian view and deduce that similar demonstrations guide the concepts of query, with consistent labels serving as estimates. Based on this understanding, we establish a label propagation framework to link label consistency with propagation error bounds. To model label consistency, we propose a data synthesis method, leveraging both semantic and label information, and use TopK sampling with Synthetic Data (TopK-SD) to acquire demonstrations with consistent labels. TopK-SD outperforms original TopK sampling on multiple benchmarks. Our work provides a new perspective for understanding the working mechanisms within ICL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 27 of its 49 references that are in the map. First posted on arXiv 13 Dec 2025.

Builds on

27 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermism5 Jan
  2. 2023How Do In-Context Examples Affect Compositional Generalization?8 May
  3. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  12. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
12 more
  1. 2022A Survey on In-context Learningcited by 717
  2. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  3. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  4. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  5. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  6. 2023Text Classification via Large Language Modelscited by 104
  7. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  8. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  9. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 57
  10. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  11. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  12. 2023CausalLM is not optimal for in-context learningcited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  2. 2025On the Relationship Between the Choice of Representation and In-Context Learning
  3. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
  5. 2023Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps
  6. 2025Refract ICL: Rethinking Example Selection in the Era of Million-Token Models
  7. 2026Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
  8. 2023Dr.ICL: Demonstration-Retrieved In-context Learning