# Benchmarking Machine Translation with Cultural Awareness

arXiv: https://arxiv.org/abs/2305.14328 · HTML: https://mlsanity.com/paper/2305.14328/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2010.00710/) (2020)
- [On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model](https://mlsanity.com/paper/2204.13509/) (2022)
- [Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation](https://mlsanity.com/paper/2302.07856/) (2023)
- [Training Neural Machine Translation To Apply Terminology Constraints](https://mlsanity.com/paper/1906.01105/) (2019)
- [Incorporating Discrete Translation Lexicons into Neural Machine Translation](https://mlsanity.com/paper/1606.02006/) (2016)
- [Domain Adaptation of Neural Machine Translation by Lexicon Induction](https://mlsanity.com/paper/1906.00376/) (2019)
- [Using natural language prompts for machine translation](https://mlsanity.com/paper/2202.11822/) (2022)
- [FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation](https://mlsanity.com/paper/2210.00193/) (2022)
- [On the Evaluation of Machine Translation for Terminology Consistency](https://mlsanity.com/paper/2106.11891/) (2021)
- [DEEP: DEnoising Entity Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2111.07393/) (2021)
- [SLING: A framework for frame semantic parsing](https://mlsanity.com/paper/1710.07032/) (2017)
- [varepsilon K'U <MASK>: Integrating Yor`ub'a cultural greetings into machine translation](https://mlsanity.com/paper/2303.17972/) (2023)
- [The ApposCorpus: A new multilingual, multi-domain dataset for factual appositive generation](https://mlsanity.com/paper/2011.03287/) (2020)
- [Translating Phrases in Neural Machine Translation](https://mlsanity.com/paper/1708.01980/) (2017)

## Built on by (most cited first, 30 of 53)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [Culture is Not Trivia: Sociocultural Theory for Cultural NLP](https://mlsanity.com/paper/2502.12057/) (2025)
- [Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings](https://mlsanity.com/paper/2309.08591/) (2023)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning](https://mlsanity.com/paper/2507.05418/) (2025)
- [R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learning](https://mlsanity.com/paper/2502.19735/) (2025)
- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Designing Efficient LLM Accelerators for Edge Devices](https://mlsanity.com/paper/2408.00462/) (2024)
- [MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs](https://mlsanity.com/paper/2505.21693/) (2025)
- [Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation](https://mlsanity.com/paper/2504.01919/) (2025)
- [Bridging Background Knowledge Gaps in Translation with Automatic Explicitation](https://mlsanity.com/paper/2312.01308/) (2023)
- [What Is Missing in Multilingual Visual Reasoning and How to Fix It](https://mlsanity.com/paper/2403.01404/) (2024)
- [An Interdisciplinary Approach to Human-Centered Machine Translation](https://mlsanity.com/paper/2506.13468/) (2025)
- [Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation](https://mlsanity.com/paper/2601.07338/) (2026)
- [DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context](https://mlsanity.com/paper/2509.17399/) (2025)
- [CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation](https://mlsanity.com/paper/2505.24456/) (2025)
- [Towards Style Alignment in Cross-Cultural Translation](https://mlsanity.com/paper/2507.00216/) (2025)
- [UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languages](https://mlsanity.com/paper/2509.21294/) (2025)
- [Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation](https://mlsanity.com/paper/2604.24361/) (2026)
- [Exploring the Translation Mechanism of Large Language Models](https://mlsanity.com/paper/2502.11806/) (2025)
- [Fine-Tuning LLMs for Low-Resource Dialect Translation: The Case of Lebanese](https://mlsanity.com/paper/2505.00114/) (2025)
- [NeoAMT: Neologism-Aware Agentic Machine Translation with Reinforcement Learning](https://mlsanity.com/paper/2601.03790/) (2026)
- [LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation](https://mlsanity.com/paper/2604.18490/) (2026)
- [Vernacular? I Barely Know Her: Challenges with Style Control and Stereotyping](https://mlsanity.com/paper/2406.12679/) (2024)
- [Building Large-Scale English-Romanian Literary Translation Resources with Open Models](https://mlsanity.com/paper/2509.07829/) (2025)
- [Benchmarking Machine Translation on Chinese Social Media Texts](https://mlsanity.com/paper/2601.22931/) (2026)
- [Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Miss](https://mlsanity.com/paper/2604.12911/) (2026)
- [Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC](https://mlsanity.com/paper/2605.25626/) (2026)

## Abstract (arXiv)

Translating culture-related content is vital for effective cross-cultural communication. However, many culture-specific items (CSIs) often lack viable translations across languages, making it challenging to collect high-quality, diverse parallel corpora with CSI annotations. This difficulty hinders the analysis of cultural awareness of machine translation (MT) systems, including traditional neural MT and the emerging MT paradigm using large language models (LLM). To address this gap, we introduce a novel parallel corpus, enriched with CSI annotations in 6 language pairs for investigating Culturally-Aware Machine Translation--CAMT. Furthermore, we design two evaluation metrics to assess CSI translations, focusing on their pragmatic translation quality. Our findings show the superior ability of LLMs over neural MTs in leveraging external cultural knowledge for translating CSIs, especially those lacking translations in the target culture.