# In-Context Learning for Text Classification with Many Labels

arXiv: https://arxiv.org/abs/2309.10954 · HTML: https://mlsanity.com/paper/2309.10954/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [Efficient Few-Shot Learning Without Prompts](https://mlsanity.com/paper/2209.11055/) (2022)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue](https://mlsanity.com/paper/2009.13570/) (2020)
- [ConvFiT: Conversational Fine-Tuning of Pretrained Language Models](https://mlsanity.com/paper/2109.10126/) (2021)

## Built on by (most cited first, 30 of 35)

- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [In-context Learning with Retrieved Demonstrations for Language Models: A Survey](https://mlsanity.com/paper/2401.11624/) (2024)
- [UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI](https://mlsanity.com/paper/2407.00106/) (2024)
- [RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content](https://mlsanity.com/paper/2406.11811/) (2024)
- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions](https://mlsanity.com/paper/2410.02028/) (2024)
- [Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models](https://mlsanity.com/paper/2406.07001/) (2024)
- [Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning](https://mlsanity.com/paper/2406.11890/) (2024)
- [Paraphrase and Aggregate with Large Language Models for Minimizing Intent Classification Errors](https://mlsanity.com/paper/2406.17163/) (2024)
- [ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks](https://mlsanity.com/paper/2507.01321/) (2025)
- [Re-examining Sexism and Misogyny Classification with Annotator Attitudes](https://mlsanity.com/paper/2410.03543/) (2024)
- [The Few-shot Dilemma: Over-prompting Large Language Models](https://mlsanity.com/paper/2509.13196/) (2025)
- [Memorization in In-Context Learning](https://mlsanity.com/paper/2408.11546/) (2024)
- [ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Origination](https://mlsanity.com/paper/2409.14285/) (2024)
- [Vulnerability of LLMs to Vertically Aligned Text Manipulations](https://mlsanity.com/paper/2410.20016/) (2024)
- [Dynamic Label Name Refinement for Few-Shot Dialogue Intent Classification](https://mlsanity.com/paper/2412.15603/) (2024)
- [GenCLS++: Pushing the Boundaries of Generative Classification in LLMs Through Comprehensive SFT and RL Studies Across Diverse Datasets](https://mlsanity.com/paper/2504.19898/) (2025)
- [Grimoire is All You Need for Enhancing Large Language Models](https://mlsanity.com/paper/2401.03385/) (2024)
- [One size doesn't fit all: Predicting the Number of Examples for In-Context Learning](https://mlsanity.com/paper/2403.06402/) (2024)
- [Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model](https://mlsanity.com/paper/2409.17745/) (2024)
- [In-Context Learning (and Unlearning) of Length Biases](https://mlsanity.com/paper/2502.06653/) (2025)
- [When LLMs are Unfit Use FastFit: Fast and Effective Text Classification with Many Classes](https://mlsanity.com/paper/2404.12365/) (2024)
- [RARe: Retrieval Augmented Retrieval with In-Context Examples](https://mlsanity.com/paper/2410.20088/) (2024)
- [Incomplete In-context Learning](https://mlsanity.com/paper/2505.07251/) (2025)
- [Mitigating Label Length Bias in Large Language Models](https://mlsanity.com/paper/2511.14385/) (2025)
- [Effective Self-Mining of In-Context Examples for Unsupervised Machine Translation with LLMs](https://mlsanity.com/paper/2410.11006/) (2024)
- [Leveraging In-Context Learning and Retrieval-Augmented Generation for Automatic Question Generation in Educational Domains](https://mlsanity.com/paper/2501.17397/) (2025)
- [From Annotation to Adaptation: Metrics, Synthetic Data, and Aspect Extraction for Aspect-Based Sentiment Analysis with Large Language Models](https://mlsanity.com/paper/2503.20715/) (2025)
- [Logit Separability-Driven Samples and Multiple Class-Related Words Selection for Advancing In-Context Learning](https://mlsanity.com/paper/2406.10908/) (2024)

## Abstract (arXiv)

In-context learning (ICL) using large language models for tasks with many labels is challenging due to the limited context window, which makes it difficult to fit a sufficient number of examples in the prompt. In this paper, we use a pre-trained dense retrieval model to bypass this limitation, giving the model only a partial view of the full label space for each inference call. Testing with recent open-source LLMs (OPT, LLaMA), we set new state of the art performance in few-shot settings for three common intent classification datasets, with no finetuning. We also surpass fine-tuned performance on fine-grained sentiment classification in certain cases. We analyze the performance across number of in-context examples and different model scales, showing that larger models are necessary to effectively and consistently make use of larger context lengths for ICL. By running several ablations, we analyze the model's use of: a) the similarity of the in-context examples to the current input, b) the semantic content of the class names, and c) the correct correspondence between examples and labels. We demonstrate that all three are needed to varying degrees depending on the domain, contrary to certain recent works.