From Haystack to Needle: Label Space Reduction for Zero-shot Classification
Not on the map. Why?
Abstract (arXiv)
We present Label Space Reduction (LSR), a novel method for improving zero-shot classification performance of Large Language Models (LLMs). LSR iteratively refines the classification label space by systematically ranking and reducing candidate classes, enabling the model to concentrate on the most relevant options. By leveraging unlabeled data with the statistical learning capabilities of data-driven models, LSR dynamically optimizes the label space representation at test time. Our experiments across seven benchmarks demonstrate that LSR improves macro-F1 scores by an average of 7.0% (up to 14.2%) with Llama-3.1-70B and 3.3% (up to 11.1%) with Claude-3.5-Sonnet compared to standard zero-shot classification baselines. To reduce the computational overhead of LSR, which requires an additional LLM call at each iteration, we propose distilling the model into a probabilistic classifier, allowing for efficient inference.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 12 Feb 2025.
Builds on
32 of 52 references.
References · 32
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
20 more
- 2019The Curious Case of Neural Text Degenerationcited by 2,881
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2018Hierarchical Neural Story Generationcited by 1,182
- 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
- 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
- 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
- 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
- 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
- 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
- 2024Large Language Models for Time Series: A Surveycited by 113
- 2018AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classificationcited by 101
- 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
- 2021LightXML: Transformer with Dynamic Negative Sampling for High-Performance Extreme Multi-label Text Classificationcited by 61
- 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
- 2023Sparks of Large Audio Models: A Survey and Outlookcited by 41
- 2018Group Preserving Label Embedding for Multi-Label Classificationcited by 7
- 2023ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classificationcited by 5
- 2023Multi-modal Extreme Classificationcited by 3
- 2024Large Language Model as a Teacher for Zero-shot Tagging at Extreme Scalescited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.