mlsanityData to 14 Oct 2026

DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation

arXiv 2503.01622 ↗First posted 3 Mar 2025cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Recent work found that LLMs are sensitive to a wide range of arbitrary prompt dimensions, including the type of delimiters, answer enumerators, instruction wording, and more. This throws into question popular single-prompt evaluation practices. We present DOVE (Dataset Of Variation Evaluation) a large-scale dataset containing prompt perturbations of various evaluation benchmarks. In contrast to previous work, we examine LLM sensitivity from an holistic perspective, and assess the joint effects of perturbations along various dimensions, resulting in thousands of perturbations per instance. We evaluate several model families against DOVE, leading to several findings, including efficient methods for choosing well-performing prompts, observing that few-shot examples reduce sensitivity, and identifying instances which are inherently hard across all perturbations. DOVE consists of more than 250M prompt perturbations and model outputs, which we make publicly available to spur a community-wide effort toward meaningful, robust, and efficient evaluation. Browse the data, contribute, and more: https://slab-nlp.github.io/DOVE/

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

26 of 43 references.

References · 26

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  8. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  9. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  10. 2021True Few-Shot Learning with Language Modelscited by 344
  11. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  12. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
14 more
  1. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  2. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
  3. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  4. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  5. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  6. 2024Efficient multi-prompt evaluation of LLMscited by 73
  7. 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
  8. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 49
  9. 2023The language of prompting: What linguistic properties make a prompt successful?cited by 44
  10. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 42
  11. 2024LLMs May Perform MCQA by Selecting the Least Incorrect Optioncited by 33
  12. 2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 25
  13. 2024Sloth: scaling laws for LLM skills to predict multi-benchmark performance across familiescited by 24
  14. 2024Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AIcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 12.

21 papers.

Sort:

By area:

  1. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  2. 2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 21
  3. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  4. 2025ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Momentscited by 11
  5. 2025LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblancecited by 9
  6. 2025Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Modelscited by 6
  7. 2026ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Modelscited by 6
  8. 2025PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generationcited by 3
  9. 2026Revisiting Prompt Sensitivity in Large Language Models for Text Classification: The Role of Prompt Underspecificationcited by 3
  10. 2026Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Resultscited by 3
  11. 2026Position: Evaluation Scores Are Perishable Knowledge Claimscited by 3
  12. 2025SCAN: Structured Capability Assessment and Navigation for LLMscited by 2
9 more
  1. 2026Robustness as an Emergent Property of Task Performancecited by 2
  2. 2025Are the LLMs Capable of Maintaining at Least the Language Genus?cited by 1
  3. 2025CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variationscited by 1
  4. 2026Instructions Shape Production of Language, not Processingcited by 1
  5. 2025Do LLMs Triage Like Clinicians? A Dynamic Study of Outpatient Referralcited by 0
  6. 2026Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluationcited by 0
  7. 2026Prompt-Robust Language Models: Which Training Strategies Work?cited by 0
  8. 2026Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indicescited by 0
  9. 2026Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMscited by 0