# Modelling Latent Translations for Cross-Lingual Transfer

arXiv: https://arxiv.org/abs/2107.11353 · HTML: https://mlsanity.com/paper/2107.11353/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 23 Jul 2021.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [A Survey Of Cross-lingual Word Embedding Models](https://mlsanity.com/paper/1706.04902/) (2017)
- [Sequence-to-Sequence Learning as Beam-Search Optimization](https://mlsanity.com/paper/1606.02960/) (2016)
- [On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation](https://mlsanity.com/paper/2005.03642/) (2020)
- [Minimum Risk Training for Neural Machine Translation](https://mlsanity.com/paper/1512.02433/) (2015)
- [Translation Artifacts in Cross-lingual Transfer Learning](https://mlsanity.com/paper/2004.04721/) (2020)
- [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://mlsanity.com/paper/1909.06694/) (2019)
- [Can Neural Machine Translation be Improved with User Feedback?](https://mlsanity.com/paper/1804.05958/) (2018)
- [Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing](https://mlsanity.com/paper/1807.00914/) (2018)
- [Classical Structured Prediction Losses for Sequence to Sequence Learning](https://mlsanity.com/paper/1711.04956/) (2017)
- [Learning to Paraphrase for Question Answering](https://mlsanity.com/paper/1708.06022/) (2017)

## Built on by (most cited first, 16 of 16)

- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [Do Multilingual Language Models Think Better in English?](https://mlsanity.com/paper/2308.01223/) (2023)
- [Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://mlsanity.com/paper/2203.09435/) (2022)
- [Revisiting Machine Translation for Cross-lingual Classification](https://mlsanity.com/paper/2305.14240/) (2023)
- [Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation](https://mlsanity.com/paper/2201.13405/) (2022)
- [Fine-tuning Large Language Models with Sequential Instructions](https://mlsanity.com/paper/2403.07794/) (2024)
- [Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese](https://mlsanity.com/paper/2304.08823/) (2023)
- [Translation-Enhanced Multilingual Text-to-Image Generation](https://mlsanity.com/paper/2305.19216/) (2023)
- [Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages](https://mlsanity.com/paper/2406.12739/) (2024)
- [T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification](https://mlsanity.com/paper/2306.04996/) (2023)
- [Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning](https://mlsanity.com/paper/2601.18722/) (2026)
- [Multilingual Safety Alignment Via Sparse Weight Editing](https://mlsanity.com/paper/2602.22554/) (2026)
- [Transfer-Free Data-Efficient Multilingual Slot Labeling](https://mlsanity.com/paper/2305.13528/) (2023)
- [Translate, then Detect: Leveraging Machine Translation for Cross-Lingual Toxicity Classification](https://mlsanity.com/paper/2509.14493/) (2025)
- [Language Ideologies in a Multilingual Society: An LLM-based Analysis of Luxembourgish News Comments](https://mlsanity.com/paper/2604.27661/) (2026)

## Abstract (arXiv)

While achieving state-of-the-art results in multiple tasks and languages, translation-based cross-lingual transfer is often overlooked in favour of massively multilingual pre-trained encoders. Arguably, this is due to its main limitations: 1) translation errors percolating to the classification phase and 2) the insufficient expressiveness of the maximum-likelihood translation. To remedy this, we propose a new technique that integrates both steps of the traditional pipeline (translation and classification) into a single model, by treating the intermediate translations as a latent random variable. As a result, 1) the neural machine translation system can be fine-tuned with a variant of Minimum Risk Training where the reward is the accuracy of the downstream task classifier. Moreover, 2) multiple samples can be drawn to approximate the expected loss across all possible translations during inference. We evaluate our novel latent translation-based model on a series of multilingual NLU tasks, including commonsense reasoning, paraphrase identification, and natural language inference. We report gains for both zero-shot and few-shot learning setups, up to 2.7 accuracy points on average, which are even more prominent for low-resource languages (e.g., Haitian Creole). Finally, we carry out in-depth analyses comparing different underlying NMT models and assessing the impact of alternative translations on the downstream performance.