# Sentiment-Aware Measure (SAM) for Evaluating Sentiment Transfer by Machine Translation Systems

arXiv: https://arxiv.org/abs/2109.14895 · HTML: https://mlsanity.com/paper/2109.14895/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [WASSA-2017 Shared Task on Emotion Intensity](https://mlsanity.com/paper/1708.03700/) (2017)
- [Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task](https://mlsanity.com/paper/2010.04297/) (2020)
- [Is it Great or Terrible? Preserving Sentiment in Neural Machine Translation of Arabic Reviews](https://mlsanity.com/paper/2010.13814/) (2020)
- [SentiWords: Deriving a High Precision and High Coverage Lexicon for Sentiment Analysis](https://mlsanity.com/paper/1510.09079/) (2015)

## Built on by (most cited first, 4 of 4)

- [A Semi-supervised Approach for a Better Translation of Sentiment in Dialectical Arabic UGT](https://mlsanity.com/paper/2210.11899/) (2022)
- [Applications and Challenges of Sentiment Analysis in Real-life Scenarios](https://mlsanity.com/paper/2301.09912/) (2023)
- [Automatically Generating Chinese Homophone Words to Probe Machine Translation Estimation Systems](https://mlsanity.com/paper/2503.16158/) (2025)
- [Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study](https://mlsanity.com/paper/2405.11668/) (2024)

## Abstract (arXiv)

In translating text where sentiment is the main message, human translators give particular attention to sentiment-carrying words. The reason is that an incorrect translation of such words would miss the fundamental aspect of the source text, i.e. the author's sentiment. In the online world, MT systems are extensively used to translate User-Generated Content (UGC) such as reviews, tweets, and social media posts, where the main message is often the author's positive or negative attitude towards the topic of the text. It is important in such scenarios to accurately measure how far an MT system can be a reliable real-life utility in transferring the correct affect message. This paper tackles an under-recognised problem in the field of machine translation evaluation which is judging to what extent automatic metrics concur with the gold standard of human evaluation for a correct translation of sentiment. We evaluate the efficacy of conventional quality metrics in spotting a mistranslation of sentiment, especially when it is the sole error in the MT output. We propose a numerical `sentiment-closeness' measure appropriate for assessing the accuracy of a translated affect message in UGC text by an MT system. We will show that incorporating this sentiment-aware measure can significantly enhance the correlation of some available quality metrics with the human judgement of an accurate translation of sentiment.