# compare-mt: A Tool for Holistic Comparison of Language Generation Systems

arXiv: https://arxiv.org/abs/1903.07926 · HTML: https://mlsanity.com/paper/1903.07926/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2019.

## Built on by (most cited first, 30 of 43)

- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [ExplainaBoard: An Explainable Leaderboard for NLP](https://mlsanity.com/paper/2104.06387/) (2021)
- [DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises](https://mlsanity.com/paper/2302.10025/) (2023)
- [When Does Translation Require Context? A Data-driven, Multilingual Exploration](https://mlsanity.com/paper/2109.07446/) (2021)
- [Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://mlsanity.com/paper/2103.02262/) (2021)
- [Towards More Fine-grained and Reliable NLP Performance Prediction](https://mlsanity.com/paper/2102.05486/) (2021)
- [CTC Alignments Improve Autoregressive Translation](https://mlsanity.com/paper/2210.05200/) (2022)
- [FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation](https://mlsanity.com/paper/2206.04294/) (2022)
- [The statistical advantage of automatic NLG metrics at the system level](https://mlsanity.com/paper/2105.12437/) (2021)
- [TranSmart: A Practical Interactive Machine Translation System](https://mlsanity.com/paper/2105.13072/) (2021)
- [Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation](https://mlsanity.com/paper/2106.00941/) (2021)
- [Meta Back-translation](https://mlsanity.com/paper/2102.07847/) (2021)
- [On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation](https://mlsanity.com/paper/2110.01811/) (2021)
- [On the Language Coverage Bias for Neural Machine Translation](https://mlsanity.com/paper/2106.03297/) (2021)
- [Simple and Scalable Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2302.12188/) (2023)
- [Angler: Helping Machine Translation Practitioners Prioritize Model Improvements](https://mlsanity.com/paper/2304.05967/) (2023)
- [MMTAfrica: Multilingual Machine Translation for African Languages](https://mlsanity.com/paper/2204.04306/) (2022)
- [Integrating Vectorized Lexical Constraints for Neural Machine Translation](https://mlsanity.com/paper/2203.12210/) (2022)
- [CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.00665/) (2022)
- [What Do You Get When You Cross Beam Search with Nucleus Sampling?](https://mlsanity.com/paper/2107.09729/) (2021)
- [Improving Retrieval Augmented Neural Machine Translation by Controlling Source and Fuzzy-Match Interactions](https://mlsanity.com/paper/2210.05047/) (2022)
- [TabGenie: A Toolkit for Table-to-Text Generation](https://mlsanity.com/paper/2302.14169/) (2023)
- [An Exploration of Data Augmentation Techniques for Improving English to Tigrinya Translation](https://mlsanity.com/paper/2103.16789/) (2021)
- [Multi-Granularity Optimization for Non-Autoregressive Translation](https://mlsanity.com/paper/2210.11017/) (2022)
- [Non-Autoregressive Neural Machine Translation: A Call for Clarity](https://mlsanity.com/paper/2205.10577/) (2022)
- [Local Byte Fusion for Neural Machine Translation](https://mlsanity.com/paper/2205.11490/) (2022)
- [Macro-Average: Rare Types Are Important Too](https://mlsanity.com/paper/2104.05700/) (2021)
- [Human-in-the-loop Machine Translation with Large Language Model](https://mlsanity.com/paper/2310.08908/) (2023)
- [Towards Accurate Translation via Semantically Appropriate Application of Lexical Constraints](https://mlsanity.com/paper/2306.12089/) (2023)
- [Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing](https://mlsanity.com/paper/2407.01394/) (2024)

## Abstract (arXiv)

In this paper, we describe compare-mt, a tool for holistic analysis and comparison of the results of systems for language generation tasks such as machine translation. The main goal of the tool is to give the user a high-level and coherent view of the salient differences between systems that can then be used to guide further analysis or system improvement. It implements a number of tools to do so, such as analysis of accuracy of generation of particular types of words, bucketed histograms of sentence accuracies or counts based on salient characteristics, and extraction of characteristic n-grams for each system. It also has a number of advanced features such as use of linguistic labels, source side data, or comparison of log likelihoods for probabilistic models, and also aims to be easily extensible by users to new types of analysis. The code is available at https://github.com/neulab/compare-mt