# Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements

arXiv: https://arxiv.org/abs/2401.06766 · HTML: https://mlsanity.com/paper/2401.06766/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 56 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Builds on (most cited first, 30 of 39)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [A critical look at the evaluation of GNNs under heterophily: Are we really making progress?](https://mlsanity.com/paper/2302.11640/) (2023)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)

## Built on by (most cited first, 30 of 83)

- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [POSIX: A Prompt Sensitivity Index For Large Language Models](https://mlsanity.com/paper/2410.02185/) (2024)
- [FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model](https://mlsanity.com/paper/2406.06004/) (2024)
- [SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature](https://mlsanity.com/paper/2406.07835/) (2024)
- [Examining the robustness of LLM evaluation to the distributional assumptions of benchmarks](https://mlsanity.com/paper/2404.16966/) (2024)
- [Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs](https://mlsanity.com/paper/2509.01790/) (2025)
- [Beemo: Benchmark of Expert-edited Machine-generated Outputs](https://mlsanity.com/paper/2411.04032/) (2024)
- [DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation](https://mlsanity.com/paper/2503.01622/) (2025)
- [Towards LLMs Robustness to Changes in Prompt Format Styles](https://mlsanity.com/paper/2504.06969/) (2025)
- [Brittlebench: Quantifying LLM robustness via prompt sensitivity](https://mlsanity.com/paper/2603.13285/) (2026)
- [The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models](https://mlsanity.com/paper/2404.05904/) (2024)
- [PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluation](https://mlsanity.com/paper/2406.18528/) (2024)
- [Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimization](https://mlsanity.com/paper/2502.04295/) (2025)
- [Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analyses](https://mlsanity.com/paper/2504.11972/) (2025)
- [Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning](https://mlsanity.com/paper/2507.22887/) (2025)
- [CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharing](https://mlsanity.com/paper/2410.16670/) (2024)
- [ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments](https://mlsanity.com/paper/2505.22169/) (2025)
- [Large Language Models Can Better Understand Knowledge Graphs Than We Thought](https://mlsanity.com/paper/2402.11541/) (2024)
- [SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Prompts](https://mlsanity.com/paper/2412.00765/) (2024)
- [When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs](https://mlsanity.com/paper/2508.11383/) (2025)
- [ContextNav: Towards Agentic Multimodal In-Context Learning](https://mlsanity.com/paper/2510.04560/) (2025)
- [Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning](https://mlsanity.com/paper/2406.11890/) (2024)
- [Token-based Decision Criteria Are Suboptimal in In-context Learning](https://mlsanity.com/paper/2406.16535/) (2024)
- [Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values](https://mlsanity.com/paper/2502.00313/) (2025)
- [LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblance](https://mlsanity.com/paper/2510.11905/) (2025)
- [SEAM: A Stochastic Benchmark for Multi-Document Tasks](https://mlsanity.com/paper/2406.16086/) (2024)
- [A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios](https://mlsanity.com/paper/2408.01963/) (2024)

## Abstract (arXiv)

Large language models demonstrate a remarkable capability for learning to solve new tasks from a few examples. The prompt template, or the way the input examples are formatted to obtain the prompt, is an important yet often overlooked aspect of in-context learning. In this work, we conduct a comprehensive study of the template format's influence on the in-context learning performance. We evaluate the impact of the prompt template across 21 models (from 770M to 70B parameters) and 4 standard classification datasets. We show that a poor choice of the template can reduce the performance of the strongest models and inference methods to a random guess level. More importantly, the best templates do not transfer between different setups and even between models of the same family. Our findings show that the currently prevalent approach to evaluation, which ignores template selection, may give misleading results due to different templates in different works. As a first step towards mitigating this issue, we propose Template Ensembles that aggregate model predictions across several templates. This simple test-time augmentation boosts average performance while being robust to the choice of random set of templates.