StaICC: Standardized Evaluation for Classification Task in In-context Learning
Not on the map. Why?
Abstract (arXiv)
Classification tasks are widely investigated in the In-Context Learning (ICL) paradigm. However, current efforts are evaluated on disjoint benchmarks and settings, while their performances are significantly influenced by some trivial variables, such as prompt templates, data sampling, instructions, etc., which leads to significant inconsistencies in the results reported across various literature, preventing fair comparison or meta-analysis across different papers. Therefore, this paper proposes a standardized and easy-to-use evaluation toolkit (StaICC) for in-context classification. Including, for the normal classification task, we provide StaICC-Normal, selecting 10 widely used datasets, and generating prompts with a fixed form, to mitigate the variance among the experiment implementations. To enrich the usage of our benchmark, we also provide a sub-benchmark StaICC-Diag for diagnosing ICL from several aspects, aiming for a more robust inference processing.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 64 of its 87 references that are in the map. First posted on arXiv 27 Jan 2025.
Builds on
64 of 87 references.
References · 64
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2023Qwen Technical Reportcited by 3,273
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2024Qwen2 Technical Reportcited by 2,076
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
38 more
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
- 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
- 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
- 2021MetaICL: Learning to Learn In Contextcited by 450
- 2023Larger language models do in-context learning differentlycited by 386
- 2021Explaining Neural Scaling Lawscited by 353
- 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
- 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
- 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
- 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
- 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
- 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
- 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
- 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
- 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
- 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
- 2023Unified Demonstration Retriever for In-Context Learningcited by 141
- 2023The Learnability of In-Context Learningcited by 126
- 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
- 2023In-Context Convergence of Transformerscited by 112
- 2023Finding Support Examples for In-Context Learningcited by 104
- 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
- 2023Mitigating Label Biases for In-context Learningcited by 88
- 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
- 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
- 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 76
- 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
- 2023Symbol tuning improves in-context learning in language modelscited by 72
- 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
- 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
- 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
- 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
- 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
50 most cited of 64 All 64 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.