Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels
Not on the map. Why?
Abstract (arXiv)
Can in-context learning (ICL) override pre-trained label semantics, or does it merely refine an existing semantic backbone? We address this question by treating LLMs as prompt-induced classifiers and contrasting their behavior under natural demonstrations (with correct labels) and inverted demonstrations (systematically flipping label meanings). We decompose ICL behavior into three alignment metrics (truth, prior, and prompt alignment) and introduce a semantic override rate, defined as correctness under flipped semantics. Across eight classification tasks and eight open-source LLMs (1--12B parameters), we find consistent evidence for a semantic anchor view. With natural demonstrations, ICL improves accuracy while maintaining strong prior alignment; most correct predictions coincide with zero-shot behavior, even when the prior is weak. With inverted demonstrations, models cannot learn coherent anti-semantic classifiers: prompt alignment increases only by sacrificing accuracy, and semantic override rates remain exactly zero in our few-shot 1--12B setting. Rather than flexibly remapping label meanings, ICL primarily adjusts how inputs project onto stable semantic directions learned during pre-training, clarifying fundamental limits of few-shot prompting and suggesting that overriding label semantics at these scales requires interventions beyond ICL. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/semantic-anchors-icl.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 26 Nov 2025.
Builds on
26 of 31 references.
References · 26
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2025Qwen3 Technical Reportcited by 8,820
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2015A large annotated corpus for learning natural language inferencecited by 1,847
- 2025Gemma 3 Technical Reportcited by 1,824
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
- 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
14 more
- 2022Transformers learn in-context by gradient descentcited by 732
- 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
- 2023Larger language models do in-context learning differentlycited by 394
- 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
- 2024Many-Shot In-Context Learningcited by 206
- 2023Mitigating Label Biases for In-context Learningcited by 89
- 2023In-Context Learning through the Bayesian Prismcited by 83
- 2023Symbol tuning improves in-context learning in language modelscited by 75
- 2020ETHOS: an Online Hate Speech Detection Datasetcited by 66
- 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 57
- 2022Prototypical Verbalizer for Prompt-based Few-shot Tuningcited by 53
- 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 51
- 2022Label Semantic Aware Pre-training for Few-shot Text Classificationcited by 18
- 2025Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decodingcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.