# Pearmut: Human Evaluation of Translation Made Trivial

arXiv: https://arxiv.org/abs/2601.02933 · HTML: https://mlsanity.com/paper/2601.02933/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 16 of 16)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers](https://mlsanity.com/paper/2106.15195/) (2021)
- [RankME: Reliable Human Ratings for Natural Language Generation](https://mlsanity.com/paper/1803.05928/) (2018)
- [POTATO: The Portable Text Annotation Tool](https://mlsanity.com/paper/2212.08620/) (2022)
- [Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies](https://mlsanity.com/paper/2401.06760/) (2024)
- [Efficient Online Scalar Annotation with Bounded Support](https://mlsanity.com/paper/1806.01170/) (2018)
- [GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation](https://mlsanity.com/paper/2101.06561/) (2021)
- [AI-Assisted Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.12419/) (2024)
- [Toward More Effective Human Evaluation for Machine Translation](https://mlsanity.com/paper/2204.05307/) (2022)
- [Enhancing Human Evaluation in Machine Translation with Comparative Judgment](https://mlsanity.com/paper/2502.17797/) (2025)
- [Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs](https://mlsanity.com/paper/2512.16378/) (2025)
- [factgenie: A Framework for Span-based Evaluation of Generated Texts](https://mlsanity.com/paper/2407.17863/) (2024)

## Built on by (most cited first, 11 of 11)

- [PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation](https://mlsanity.com/paper/2601.18006/) (2026)
- [Translating Under Pressure: Domain-Aware LLMs for Crisis Communication](https://mlsanity.com/paper/2604.26597/) (2026)
- [Contrastive ESA: Human Evaluation of Multiple Translations at Once](https://mlsanity.com/paper/2607.26640/) (2026)
- [Dynamically Allocating Evaluation Effort for Model Ranking](https://mlsanity.com/paper/2608.03437/) (2026)
- [Augmenting Text to Increase Translation Difficulty](https://mlsanity.com/paper/2608.15932/) (2026)
- [Last Translation Benchmark](https://mlsanity.com/paper/2609.04173/) (2026)
- [AI-assisted cultural heritage dissemination: Comparing NMT and glossary-augmented LLM translation in rock art documents](https://mlsanity.com/paper/2605.14679/) (2026)
- [The Role of Disfluencies in Speech Translation](https://mlsanity.com/paper/2608.02138/) (2026)
- [The Blindness of Document-Level Translation Evaluation](https://mlsanity.com/paper/2609.05949/) (2026)
- [Generating Adversarial Texts for Machine Translation via GRPO](https://mlsanity.com/paper/2609.06048/) (2026)
- [Controlling Backchannels in Streamable Full-duplex Models](https://mlsanity.com/paper/2609.29418/) (2026)

## Abstract (arXiv)

Human evaluation is the gold standard for multilingual NLP, but is often skipped in practice and substituted with automatic metrics because it is notoriously complex and slow to set up with existing tools with substantial engineering and operational overhead. We introduce Pearmut, a lightweight yet feature-rich platform that makes end-to-end human evaluation as easy to run as automatic evaluation. Pearmut removes common entry barriers and provides support for evaluating multilingual tasks, with a particular focus on machine translation. The platform implements standard evaluation protocols, including DA, ESA, and MQM, and is extensible to support new protocols. It features document-level context, absolute and contrastive evaluation, attention checks, ESAAI pre-annotations and both static and dynamic assignment strategies. Pearmut enables reliable human evaluation to become a practical, routine component of model development and diagnosis rather than an occasional effort.