mlsanityData to 14 Oct 2026

Token-based Decision Criteria Are Suboptimal in In-context Learning

arXiv 2406.16535 ↗First posted 24 Jun 2024cs.CL cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

In-Context Learning (ICL) typically utilizes classification criteria from output probabilities of manually selected label tokens. However, we argue that such token-based classification criteria lead to suboptimal decision boundaries, despite delicate calibrations through translation and constrained rotation applied. To address this problem, we propose Hidden Calibration, which renounces token probabilities and uses the nearest centroid classifier on the LM's last hidden states. In detail, we assign the label of the nearest centroid previously estimated from a calibration set to the test sample as the predicted label. Our experiments on 6 models and 10 classification datasets indicate that Hidden Calibration consistently outperforms current token-based baselines by about 20%~50%, achieving a strong state-of-the-art in ICL. Our further analysis demonstrates that Hidden Calibration finds better classification criteria with less inter-class overlap, and LMs provide linearly separable intra-class clusters with the help of demonstrations, which supports Hidden Calibration and gives new insights into the principle of ICL. Our official code implementation can be found at https://github.com/hc495/Hidden_Calibration.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

37 of 57 references.

References · 37

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  9. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  10. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  11. 2022A Survey on In-context Learningcited by 721
  12. 2021MetaICL: Learning to Learn In Contextcited by 454
25 more
  1. 2023In-Context Learning Creates Task Vectorscited by 312
  2. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  3. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  4. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  5. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  6. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 210
  7. 2018Hate Speech Dataset from a White Supremacy Forumcited by 198
  8. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  9. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  10. 2023Mitigating Label Biases for In-context Learningcited by 89
  11. 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
  12. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  13. 2023Symbol tuning improves in-context learning in language modelscited by 75
  14. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generatorcited by 68
  15. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  16. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 57
  17. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 48
  18. 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
  19. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  20. 2023Pre-Training to Learn in Contextcited by 42
  21. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  22. 2023Generative Calibration for In-context Learningcited by 20
  23. 2024Enhancing In-context Learning via Linear Probe Calibrationcited by 16
  24. 2024In-Context Example Ordering Guided by Label Distributionscited by 12
  25. 2024NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 17
  2. 2025Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learningcited by 10
  3. 2025StaICC: Standardized Evaluation for Classification Task in In-context Learningcited by 2
  4. 2025Stability of In-Context Learning: A Spectral Coverage Perspectivecited by 2
  5. 2025Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysiscited by 2
  6. 2025Boosting In-Context Learning in LLMs Through the Lens of Classical Supervised Learningcited by 1
  7. 2025Mechanistic Fine-tuning for In-context Learningcited by 0
  8. 2026Self-Improving In-Context Learningcited by 0
  9. 2026Are In-Context Images Worth 10 Dimensions?cited by 0