mlsanityData to 14 Oct 2026

Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild

arXiv 2605.24213 ↗First posted 22 May 2026cs.SE cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Evaluation harnesses are software systems that orchestrate model evaluation by managing model invocation, data loading, metric computation, and result reporting. Despite their critical role in machine learning infrastructure, their operational challenges and engineering concerns have received limited attention so far. We present an empirical study of 57 evaluation harnesses, deriving a five-stage harness model and classifying 16,560 issues by workflow stage and root cause. Most harness operational challenges concentrate in the Specification stage (41.4% of issues), where harnesses integrate external models, datasets, and scoring judges. The three most frequent root causes of operational challenges are unimplemented features (24.3%), documentation gaps (20.3%), and missing input validation (17.2%), which together account for 61.7% of classified issues, spanning both defects in existing functionality and capability gaps that block intended workflows. Root causes also vary by workflow stage: environment incompatibility and external dependency breakage account for 36.2% of provisioning issues, whereas algorithmic error (25.9%) and validation gap (22.5%) dominate assessment issues. Together, these contributions establish an empirical foundation for treating evaluation engineering as a distinct software engineering concern.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 57 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

15 of 57 references.

References · 15

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2018Datasheets for Datasetscited by 1,386
  4. 2018Model Cards for Model Reportingcited by 1,080
  5. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  6. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  7. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
  8. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  9. 2020Challenges in Deploying Machine Learning: a Survey of Case Studiescited by 152
  10. 2021The Benchmark Lotterycited by 136
  11. 2022Machine Learning Operations (MLOps): Overview, Definition, and Architecturecited by 128
  12. 2024Reproducibility in Machine Learning-based Research: Overview, Barriers and Driverscited by 34
3 more
  1. 2024Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Designcited by 13
  2. 2023An Empirical Study of Self-Admitted Technical Debt in Machine Learning Softwarecited by 5
  3. 2025Pinning Is Futile: You Need More Than Local Dependency Versioning to Defend against Supply Chain Attackscited by 4

Built on by

Not cited yet.