Data to 14 Oct 2026

Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements

arXiv 2401.06766 ↗First posted 12 Jan 2024cs.CLcited by 83

Not on the map. Why?

Abstract (arXiv)

Large language models demonstrate a remarkable capability for learning to solve new tasks from a few examples. The prompt template, or the way the input examples are formatted to obtain the prompt, is an important yet often overlooked aspect of in-context learning. In this work, we conduct a comprehensive study of the template format's influence on the in-context learning performance. We evaluate the impact of the prompt template across 21 models (from 770M to 70B parameters) and 4 standard classification datasets. We show that a poor choice of the template can reduce the performance of the strongest models and inference methods to a random guess level. More importantly, the best templates do not transfer between different setups and even between models of the same family. Our findings show that the currently prevalent approach to evaluation, which ignores template selection, may give misleading results due to different templates in different works. As a first step towards mitigating this issue, we propose Template Ensembles that aggregate model predictions across several templates. This simple test-time augmentation boosts average performance while being robust to the choice of random set of templates.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

39 of 56 references.

References · 39

2020Language Models are Few-Shot Learnerscited by 32,419
2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
2015Character-level Convolutional Networks for Text Classificationcited by 2,330
2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434

27 more

2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
2019How Can We Know What Language Models Know?cited by 959
2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
2021MetaICL: Learning to Learn In Contextcited by 454
2023Larger language models do in-context learning differentlycited by 394
2023A critical look at the evaluation of GNNs under heterophily: Are we really making progress?cited by 343
2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
2020A Metric Learning Reality Checkcited by 321
2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
2019Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approachescited by 185
2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
2023Mitigating Label Biases for In-context Learningcited by 89
2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
2023In-context Example Selection with Influencescited by 61
2022Data Curation Alone Can Stabilize In-context Learningcited by 59
2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
2023The language of prompting: What linguistic properties make a prompt successful?cited by 44
2022PromptBoosting: Black-Box Text Classification with Ten Forward Passescited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 29; 2025: 34; 2026: 20.

83 papers.

Sort:

By area:

2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
2024Efficient multi-prompt evaluation of LLMscited by 73
2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 45
2024SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literaturecited by 41
2024Examining the robustness of LLM evaluation to the distributional assumptions of benchmarkscited by 31
2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMscited by 30
2024Beemo: Benchmark of Expert-edited Machine-generated Outputscited by 27
2025DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluationcited by 21
2025Towards LLMs Robustness to Changes in Prompt Format Stylescited by 21

38 more

2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 21
2024The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Modelscited by 17
2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 16
2025Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimizationcited by 16
2025Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analysescited by 14
2025Where to show Demos in Your Prompt: A Positional Bias of In-Context Learningcited by 13
2024CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharingcited by 12
2025ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Momentscited by 11
2024Large Language Models Can Better Understand Knowledge Graphs Than We Thoughtcited by 10
2024SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Promptscited by 10
2025When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMscited by 10
2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 10
2024Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learningcited by 9
2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
2025Distributive Fairness in Large Language Models: Evaluating Alignment with Human Valuescited by 9
2025LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblancecited by 9
2024SEAM: A Stochastic Benchmark for Multi-Document Taskscited by 8
2024A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarioscited by 8
2025NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmarkcited by 8
2026When Wording Steers the Evaluation: Framing Bias in LLM judgescited by 8
2026Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetrycited by 8
2024SSE: Multimodal Semantic Data Selection and Enrichment for Industrial-scale Data Assimilationcited by 7
2025Linguistic Characteristics of AI-Generated Text: A Surveycited by 7
2024When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Modelscited by 6
2024Mixtures of In-Context Learnerscited by 5
2025Human Preferences in Large Language Model Latent Space: A Technical Analysis on the Reliability of Synthetic Data in Voting Outcome Predictioncited by 5
2025Same Question, Different Words: A Latent Adversarial Framework for Prompt Robustnesscited by 5
2024Stronger Random Baselines for In-Context Learningcited by 4
2024Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methodscited by 4
2025Unlocking In-Context Learning for Natural Datasets Beyond Language Modellingcited by 4
2026Measuring Pragmatic Influence in Large Language Model Instructionscited by 4
2026On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecificationcited by 4
2024KGPA: Robustness Evaluation for Large Language Models via Cross-Domain Knowledge Graphscited by 3
2024A Survey on Automatic Credibility Assessment Using Textual Credibility Signals in the Era of Large Language Modelscited by 3
2025Cooking Up Creativity: Enhancing LLM Creativity through Structured Recombinationcited by 3
2025Leveraging LLM Inconsistency to Boost Pass@k Performancecited by 3
2025PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Modelscited by 3
2025Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMscited by 3

50 most cited of 83 All 83 (JSON)