mlsanityData to 7 Oct 2026

DEEP: Docker-based Execution and Evaluation Platform

arXiv 2602.19583 ↗First posted 23 Feb 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Comparative evaluation of several systems is a recurrent task in researching. It is a key step before deciding which system to use for our work, or, once our research has been conducted, to demonstrate the potential of the resulting model. Furthermore, it is the main task of competitive, public challenges evaluation. Our proposed software (DEEP) automates both the execution and scoring of machine translation and optical character recognition models. Furthermore, it is easily extensible to other tasks. DEEP is prepared to receive dockerized systems, run them (extracting information at that same time), and assess hypothesis against some references. With this approach, evaluators can achieve a better understanding of the performance of each model. Moreover, the software uses a clustering algorithm based on a statistical analysis of the significance of the results yielded by each model, according to the evaluation metrics. As a result, evaluators are able to identify clusters of performance among the swarm of proposals and have a better understanding of the significance of their differences. Additionally, we offer a visualization web-app to ensure that the results can be adequately understood and interpreted. Finally, we present an exemplary case of use of DEEP.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 45 references that are in the map. First posted on arXiv 23 Feb 2026.

Builds on

19 of 45 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  8. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  9. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  10. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  11. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  12. 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
7 more
  1. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  2. 2025Reinforcement Learning from Human Feedbackcited by 53
  3. 2022deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networkscited by 28
  4. 2025Seed-X: Building Strong Multilingual Translation LLM with 7B Parameterscited by 21
  5. 2025Toward Holistic Evaluation of Recommender Systems Powered by Generative Modelscited by 11
  6. 2023End-to-End Page-Level Assessment of Handwritten Text Recognitioncited by 8
  7. 2023Inspecting Explainability of Transformer Models with Additional Statistical Informationcited by 2

Built on by

Not cited yet.