# Hacking Neural Evaluation Metrics with Single Hub Text

arXiv: https://arxiv.org/abs/2512.16323 · HTML: https://mlsanity.com/paper/2512.16323/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Builds on (most cited first, 13 of 13)

- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Text Embeddings Reveal (Almost) As Much As Text](https://mlsanity.com/paper/2310.06816/) (2023)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://mlsanity.com/paper/2005.10283/) (2020)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [Improving zero-shot learning by mitigating the hubness problem](https://mlsanity.com/paper/1412.6568/) (2014)
- [Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks](https://mlsanity.com/paper/2310.11612/) (2023)
- [There's no Data Like Better Data: Using QE Metrics for MT Data Filtering](https://mlsanity.com/paper/2311.05350/) (2023)
- [JParaCrawl v3.0: A Large-scale English-Japanese Parallel Corpus](https://mlsanity.com/paper/2202.12607/) (2022)
- [Adversarial Hubness in Multi-Modal Retrieval](https://mlsanity.com/paper/2412.14113/) (2024)
- [Nearest Neighbor Normalization Improves Multimodal Retrieval](https://mlsanity.com/paper/2410.24114/) (2024)

## Built on by (most cited first, 1 of 1)

- [One Single Hub Text Breaks CLIP: Identifying Vulnerabilities in Cross-Modal Encoders via Hubness](https://mlsanity.com/paper/2604.27674/) (2026)

## Abstract (arXiv)

Strongly human-correlated evaluation metrics serve as an essential compass for the development and improvement of generation models and must be highly reliable and robust. Recent embedding-based neural text evaluation metrics, such as COMET for translation tasks, are widely used in both research and development fields. However, there is no guarantee that they yield reliable evaluation results due to the black-box nature of neural networks. To raise concerns about the reliability and safety of such metrics, we propose a method for finding a single adversarial text in the discrete space that is consistently evaluated as high-quality, regardless of the test cases, to identify the vulnerabilities in evaluation metrics. The single hub text found with our method achieved 79.1 COMET% and 67.8 COMET% in the WMT'24 English-to-Japanese (En--Ja) and English-to-German (En--De) translation tasks, respectively, outperforming translations generated individually for each source sentence by using M2M100, a general translation model. Furthermore, we also confirmed that the hub text found with our method generalizes across multiple language pairs such as Ja--En and De--En.