# EvalCards: A Framework for Standardized Evaluation Reporting

arXiv: https://arxiv.org/abs/2511.21695 · HTML: https://mlsanity.com/paper/2511.21695/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 90 references that are in the map.
- First posted on arXiv 5 Nov 2025.

## Builds on (most cited first, 30 of 32)

- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference](https://mlsanity.com/paper/2310.03003/) (2023)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](https://mlsanity.com/paper/2410.12405/) (2024)
- [BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices](https://mlsanity.com/paper/2411.12990/) (2024)
- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations](https://mlsanity.com/paper/2407.04069/) (2024)
- [Demystifying the Draft EU Artificial Intelligence Act](https://mlsanity.com/paper/2107.03721/) (2021)
- [Interactive Model Cards: A Human-Centered Approach to Model Documentation](https://mlsanity.com/paper/2205.02894/) (2022)
- [POSIX: A Prompt Sensitivity Index For Large Language Models](https://mlsanity.com/paper/2410.02185/) (2024)
- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [ECBD: Evidence-Centered Benchmark Design for NLP](https://mlsanity.com/paper/2406.08723/) (2024)
- [What's the Meaning of Superhuman Performance in Today's NLU?](https://mlsanity.com/paper/2305.08414/) (2023)
- [Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustness](https://mlsanity.com/paper/2408.04585/) (2024)
- [Towards Climate Awareness in NLP Research](https://mlsanity.com/paper/2205.05071/) (2022)
- [Machine Learning Data Practices through a Data Curation Lens: An Evaluation Framework](https://mlsanity.com/paper/2405.02703/) (2024)
- [BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks](https://mlsanity.com/paper/2410.12974/) (2024)
- [Automatic Generation of Model and Data Cards: A Step Towards Responsible AI](https://mlsanity.com/paper/2405.06258/) (2024)
- [SPHERE: An Evaluation Card for Human-AI Systems](https://mlsanity.com/paper/2504.07971/) (2025)

## Built on by (most cited first, 4 of 4)

- [The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems](https://mlsanity.com/paper/2602.17753/) (2026)
- [Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges](https://mlsanity.com/paper/2605.06161/) (2026)
- [Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation](https://mlsanity.com/paper/2608.00794/) (2026)
- [Healthcare LLM Benchmarks Are Only as Good as Their Explicit Assumptions](https://mlsanity.com/paper/2605.22612/) (2026)

## Abstract (arXiv)

Evaluation has long been a central concern in NLP, and transparent reporting practices are more critical than ever in today's landscape of rapidly released open-access models. Drawing on a survey of recent work on evaluation and documentation, we identify three persistent shortcomings in current reporting practices: reproducibility, accessibility, and governance. We argue that existing standardization efforts remain insufficient and introduce Evaluation Disclosure Cards (EvalCards) as a path forward. EvalCards are designed to enhance transparency for both researchers and practitioners while providing a practical foundation to meet emerging governance requirements.