mlsanityData to 13 Oct 2026

Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates

arXiv 2408.12263 ↗First posted 22 Aug 2024cs.CL cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

The natural language understanding (NLU) performance of large language models (LLMs) has been evaluated across various tasks and datasets. The existing evaluation methods, however, do not take into account the variance in scores due to differences in prompts, which leads to unfair evaluation and comparison of NLU performance. Moreover, evaluation designed for specific prompts is inappropriate for instruction tuning, which aims to perform well with any prompt. It is therefore necessary to find a way to measure NLU performance in a fair manner, considering score variance between different instruction templates. In this study, we provide English and Japanese cross-lingual datasets for evaluating the NLU performance of LLMs, which include multiple instruction templates for fair evaluation of each task, along with regular expressions to constrain the output format. Furthermore, we propose the Sharpe score as an evaluation metric that takes into account the variance in scores between templates. Comprehensive analysis of English and Japanese LLMs reveals that the high variance among templates has a significant impact on the fair evaluation of LLMs.

Key facts. Cited by 7 papers in the map (data through 13 Oct 2026). Builds on 47 of its 82 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

47 of 82 references.

References · 47

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,676
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,805
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,785
35 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,376
  3. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,281
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,126
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,120
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,054
  7. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,031
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2018Neural Network Acceptability Judgmentscited by 957
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 896
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 867
  12. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 797
  13. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 736
  14. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 731
  15. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 717
  16. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 705
  17. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 654
  18. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 630
  19. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 363
  20. 2023Trained Transformers Learn Linear Models In-Contextcited by 333
  21. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 249
  22. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
  23. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  24. 2020The Multilingual Amazon Reviews Corpuscited by 143
  25. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 141
  26. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  27. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  28. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 91
  29. 2021Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Modelscited by 76
  30. 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
  31. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 69
  32. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 67
  33. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  34. 2023Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuningcited by 18
  35. 2023JCoLA: Japanese Corpus of Linguistic Acceptabilitycited by 11

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 3.

7 papers.

  1. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 32
  2. 2025Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Abilitycited by 10
  3. 2024Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Modelscited by 7
  4. 2024BQA: Body Language Question Answering Dataset for Video Large Language Modelscited by 3
  5. 2026Edit-level Majority Voting Mitigates Over-Correction in LLM-based Grammatical Error Correctioncited by 1
  6. 2026Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Formscited by 0
  7. 2026Understanding Evaluation Illusion in Diffusion Large Language Modelscited by 0