# BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text

arXiv: https://arxiv.org/abs/2109.14250 · HTML: https://mlsanity.com/paper/2109.14250/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 22 references that are in the map.
- First posted on arXiv 29 Sep 2021.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)](https://mlsanity.com/paper/2006.07235/) (2020)
- [OpenKiwi: An Open Source Framework for Quality Estimation](https://mlsanity.com/paper/1902.08646/) (2019)
- [WASSA-2017 Shared Task on Emotion Intensity](https://mlsanity.com/paper/1708.03700/) (2017)
- [Is it Great or Terrible? Preserving Sentiment in Neural Machine Translation of Arabic Reviews](https://mlsanity.com/paper/2010.13814/) (2020)
- [Challenges in Translation of Emotions in Multilingual User-Generated Content: Twitter as a Case Study](https://mlsanity.com/paper/2106.10719/) (2021)

## Built on by (most cited first, 22 of 22)

- [DEBATE: Devil's Advocate-Based Assessment and Text Evaluation](https://mlsanity.com/paper/2405.09935/) (2024)
- [Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set](https://mlsanity.com/paper/2312.12624/) (2023)
- [What's under the hood: Investigating Automatic Metrics on Meeting Summarization](https://mlsanity.com/paper/2404.11124/) (2024)
- [MAU-GPT: Enhancing Multi-type Industrial Anomaly Understanding via Anomaly-aware and Generalist Experts Adaptation](https://mlsanity.com/paper/2602.07011/) (2026)
- [Weighted Grouped Query Attention in Transformers](https://mlsanity.com/paper/2407.10855/) (2024)
- [When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications](https://mlsanity.com/paper/2211.08228/) (2022)
- [TeXBLEU: Automatic Metric for Evaluate LaTeX Format](https://mlsanity.com/paper/2409.06639/) (2024)
- [A Semi-supervised Approach for a Better Translation of Sentiment in Dialectical Arabic UGT](https://mlsanity.com/paper/2210.11899/) (2022)
- [The Impact of Syntactic and Semantic Proximity on Machine Translation with Back-Translation](https://mlsanity.com/paper/2403.18031/) (2024)
- [Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translations](https://mlsanity.com/paper/2605.13596/) (2026)
- [BiVert: Bidirectional Vocabulary Evaluation using Relations for Machine Translation](https://mlsanity.com/paper/2403.03521/) (2024)
- [Measuring text summarization factuality using atomic facts entailment metrics in the context of retrieval augmented generation](https://mlsanity.com/paper/2408.15171/) (2024)
- [Context-Informed Machine Translation of Manga using Multimodal Large Language Models](https://mlsanity.com/paper/2411.02589/) (2024)
- [A Query-Aware Multi-Path Knowledge Graph Fusion Approach for Enhancing Retrieval-Augmented Generation in Large Language Models](https://mlsanity.com/paper/2507.16826/) (2025)
- [LaajMeter: A Framework for LaaJ Evaluation](https://mlsanity.com/paper/2508.10161/) (2025)
- [Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study](https://mlsanity.com/paper/2405.11668/) (2024)
- [A Deep Learning Approach for Augmenting Perceptional Understanding of Histopathology Images](https://mlsanity.com/paper/2503.06894/) (2025)
- [Unsupervised Evaluation of Multi-Turn Objective-Driven Interactions](https://mlsanity.com/paper/2511.03047/) (2025)
- [PolicyBot - Reliable Question Answering over Policy Documents](https://mlsanity.com/paper/2511.13489/) (2025)
- [Ensemble Self-Training for Unsupervised Machine Translation](https://mlsanity.com/paper/2603.17087/) (2026)
- [Not All That Is Fluent Is Factual: Investigating Hallucinations of Large Language Models in Academic Writing](https://mlsanity.com/paper/2605.04171/) (2026)
- [TAPR: Enhancing LLM Performance with a Task-Aware Prompt Rewriter](https://mlsanity.com/paper/2607.28657/) (2026)

## Abstract (arXiv)

Social media companies as well as authorities make extensive use of artificial intelligence (AI) tools to monitor postings of hate speech, celebrations of violence or profanity. Since AI software requires massive volumes of data to train computers, Machine Translation (MT) of the online content is commonly used to process posts written in several languages and hence augment the data needed for training. However, MT mistakes are a regular occurrence when translating sentiment-oriented user-generated content (UGC), especially when a low-resource language is involved. The adequacy of the whole process relies on the assumption that the evaluation metrics used give a reliable indication of the quality of the translation. In this paper, we assess the ability of automatic quality metrics to detect critical machine translation errors which can cause serious misunderstanding of the affect message. We compare the performance of three canonical metrics on meaningless translations where the semantic content is seriously impaired as compared to meaningful translations with a critical error which exclusively distorts the sentiment of the source text. We conclude that there is a need for fine-tuning of automatic metrics to make them more robust in detecting sentiment critical errors.