mlsanityData to 7 Oct 2026

EvalCards: A Framework for Standardized Evaluation Reporting

arXiv 2511.21695 ↗First posted 5 Nov 2025cs.CL cs.AI cs.CYcited by 4

Not on the map. Why?

Abstract (arXiv)

Evaluation has long been a central concern in NLP, and transparent reporting practices are more critical than ever in today's landscape of rapidly released open-access models. Drawing on a survey of recent work on evaluation and documentation, we identify three persistent shortcomings in current reporting practices: reproducibility, accessibility, and governance. We argue that existing standardization efforts remain insufficient and introduce Evaluation Disclosure Cards (EvalCards) as a path forward. EvalCards are designed to enhance transparency for both researchers and practitioners while providing a practical foundation to meet emerging governance requirements.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 32 of its 90 references that are in the map. First posted on arXiv 5 Nov 2025.

Builds on

32 of 90 references.

References · 32

  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2018Datasheets for Datasetscited by 1,371
  3. 2018Model Cards for Model Reportingcited by 1,068
  4. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  6. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  7. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
  8. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  9. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  10. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  11. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  12. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
20 more
  1. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  2. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 164
  3. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 145
  4. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  5. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  6. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  7. 2021Demystifying the Draft EU Artificial Intelligence Actcited by 82
  8. 2022Interactive Model Cards: A Human-Centered Approach to Model Documentationcited by 61
  9. 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 56
  10. 2025Toward an Evaluation Science for Generative AI Systemscited by 42
  11. 2024ECBD: Evidence-Centered Benchmark Design for NLPcited by 27
  12. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 22
  13. 2024Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustnesscited by 22
  14. 2022Towards Climate Awareness in NLP Researchcited by 19
  15. 2024Machine Learning Data Practices through a Data Curation Lens: An Evaluation Frameworkcited by 18
  16. 2024Automatic Generation of Model and Data Cards: A Step Towards Responsible AIcited by 9
  17. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  18. 2024BenchmarkCards: Standardized Documentation for Large Language Model Benchmarkscited by 8
  19. 2024On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboardscited by 6
  20. 2024Building Better Datasets: Seven Recommendations for Responsible Design from Dataset Creatorscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systemscited by 27
  2. 2026Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judgescited by 8
  3. 2026Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptionscited by 0
  4. 2026Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluationcited by 0