mlsanityData to 7 Oct 2026

The ICL Consistency Test

arXiv 2312.04945 ↗First posted 8 Dec 2023cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Just like the previous generation of task-tuned models, large language models (LLMs) that are adapted to tasks via prompt-based methods like in-context-learning (ICL) perform well in some setups but not in others. This lack of consistency in prompt-based learning hints at a lack of robust generalisation. We here introduce the ICL consistency test -- a contribution to the GenBench collaborative benchmark task (CBT) -- which evaluates how consistent a model makes predictions across many different setups while using the same data. The test is based on different established natural language inference tasks. We provide preprocessed data constituting 96 different 'setups' and a metric that estimates model consistency across these setups. The metric is provided on a fine-grained level to understand what properties of a setup render predictions unstable and on an aggregated level to compare overall model consistency. We conduct an empirical analysis of eight state-of-the-art models, and our consistency metric reveals how all tested LLMs lack robust generalisation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 22 of its 34 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

22 of 34 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
10 more
  1. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  2. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  3. 2023Larger language models do in-context learning differentlycited by 386
  4. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  5. 2022Prompting GPT-3 To Be Reliablecited by 258
  6. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  7. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  8. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  9. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  10. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 1.

6 papers.

  1. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  2. 2024Can large language models explore in-context?cited by 75
  3. 2024Efficient multi-prompt evaluation of LLMscited by 72
  4. 2024Document-Level Event Extraction with Definition-Driven ICLcited by 2
  5. 2024Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Taskscited by 0
  6. 2025ICL Optimized Fragilitycited by 0