# SALTED: A Framework for SAlient Long-Tail Translation Error Detection

arXiv: https://arxiv.org/abs/2205.09988 · HTML: https://mlsanity.com/paper/2205.09988/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 20 May 2022.

## Builds on (most cited first, 15 of 15)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [A Universal Law of Robustness via Isoperimetry](https://mlsanity.com/paper/2105.12806/) (2021)
- [Analyzing Uncertainty in Neural Machine Translation](https://mlsanity.com/paper/1803.00047/) (2018)
- [Automatic Testing and Improvement of Machine Translation](https://mlsanity.com/paper/1910.02688/) (2019)
- [Structure-Invariant Testing for Machine Translation](https://mlsanity.com/paper/1907.08710/) (2019)
- [Testing Machine Translation via Referential Transparency](https://mlsanity.com/paper/2004.10361/) (2020)
- [As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation](https://mlsanity.com/paper/2107.08357/) (2021)

## Built on by (most cited first, 23 of 23)

- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better](https://mlsanity.com/paper/2212.08597/) (2022)
- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2208.05309/) (2022)
- [Leveraging GPT-4 for Automatic Translation Post-Editing](https://mlsanity.com/paper/2305.14878/) (2023)
- [Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding](https://mlsanity.com/paper/2309.07098/) (2023)
- [Do GPTs Produce Less Literal Translations?](https://mlsanity.com/paper/2305.16806/) (2023)
- [The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics](https://mlsanity.com/paper/2305.11806/) (2023)
- [Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation](https://mlsanity.com/paper/2212.09631/) (2022)
- [Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models](https://mlsanity.com/paper/2407.16470/) (2024)
- [Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks](https://mlsanity.com/paper/2210.12929/) (2022)
- [Rank-One Editing of Encoder-Decoder Models](https://mlsanity.com/paper/2211.13317/) (2022)
- [Mitigating Hallucinated Translations in Large Language Models with Hallucination-focused Preference Optimization](https://mlsanity.com/paper/2501.17295/) (2025)
- [Investigating Multi-Pivot Ensembling with Massively Multilingual Machine Translation Models](https://mlsanity.com/paper/2311.07439/) (2023)
- [SCALE: Synergized Collaboration of Asymmetric Language Translation Engines](https://mlsanity.com/paper/2309.17061/) (2023)
- [When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation](https://mlsanity.com/paper/2601.20858/) (2026)
- [Automating Behavioral Testing in Machine Translation](https://mlsanity.com/paper/2309.02553/) (2023)
- [Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2502.01491/) (2025)
- [Operationalizing Specifications, In Addition to Test Sets for Evaluating Constrained Generative Models](https://mlsanity.com/paper/2212.00006/) (2022)
- [Towards General Error Diagnosis via Behavioral Testing in Machine Translation](https://mlsanity.com/paper/2310.13362/) (2023)
- [Fabricator or dynamic translator?](https://mlsanity.com/paper/2604.15165/) (2026)

## Abstract (arXiv)

Traditional machine translation (MT) metrics provide an average measure of translation quality that is insensitive to the long tail of behavioral problems in MT. Examples include translation of numbers, physical units, dropped content and hallucinations. These errors, which occur rarely and unpredictably in Neural Machine Translation (NMT), greatly undermine the reliability of state-of-the-art MT systems. Consequently, it is important to have visibility into these problems during model development. Towards this direction, we introduce SALTED, a specifications-based framework for behavioral testing of MT models that provides fine-grained views of salient long-tail errors, permitting trustworthy visibility into previously invisible problems. At the core of our approach is the development of high-precision detectors that flag errors (or alternatively, verify output correctness) between a source sentence and a system output. We demonstrate that such detectors could be used not just to identify salient long-tail errors in MT systems, but also for higher-recall filtering of the training data, fixing targeted errors with model fine-tuning in NMT and generating novel data for metamorphic testing to elicit further bugs in models.