# Evaluating and Improving the Coreference Capabilities of Machine Translation Models

arXiv: https://arxiv.org/abs/2302.08464 · HTML: https://mlsanity.com/paper/2302.08464/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Builds on (most cited first, 27 of 27)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](https://mlsanity.com/paper/1907.10641/) (2019)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [Getting Gender Right in Neural Machine Translation](https://mlsanity.com/paper/1909.05088/) (2019)
- [Evaluating Discourse Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1711.00513/) (2017)
- [Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem](https://mlsanity.com/paper/2004.04498/) (2020)
- [Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation](https://mlsanity.com/paper/2109.03858/) (2021)
- [Coreference Resolution without Span Representations](https://mlsanity.com/paper/2101.00434/) (2021)
- [How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs](https://mlsanity.com/paper/1612.04629/) (2016)
- [Filling Gender & Number Gaps in Neural Machine Translation with Black-box Context Injection](https://mlsanity.com/paper/1903.03467/) (2019)
- [LingMess: Linguistically Informed Multi Expert Scorers for Coreference Resolution](https://mlsanity.com/paper/2205.12644/) (2022)
- [Automatic Reference-Based Evaluation of Pronoun Translation Misses the Point](https://mlsanity.com/paper/1808.04164/) (2018)
- [Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation](https://mlsanity.com/paper/2006.08881/) (2020)
- [Yall should read this! Identifying Plurality in Second-Person Personal Pronouns in English Texts](https://mlsanity.com/paper/1910.11966/) (2019)

## Built on by (most cited first, 2 of 2)

- [Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends](https://mlsanity.com/paper/2407.21489/) (2024)
- [Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation](https://mlsanity.com/paper/2406.00787/) (2024)

## Abstract (arXiv)

Machine translation (MT) requires a wide range of linguistic capabilities, which current end-to-end models are expected to learn implicitly by observing aligned sentences in bilingual corpora. In this work, we ask: How well do MT models learn coreference resolution from implicit signal? To answer this question, we develop an evaluation methodology that derives coreference clusters from MT output and evaluates them without requiring annotations in the target language. We further evaluate several prominent open-source and commercial MT systems, translating from English to six target languages, and compare them to state-of-the-art coreference resolvers on three challenging benchmarks. Our results show that the monolingual resolvers greatly outperform MT models. Motivated by this result, we experiment with different methods for incorporating the output of coreference resolution models in MT, showing improvement over strong baselines.