# Evaluating Robustness to Input Perturbations for Neural Machine Translation

arXiv: https://arxiv.org/abs/2005.00580 · HTML: https://mlsanity.com/paper/2005.00580/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2020.

## Built on by (most cited first, 30 of 36)

- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)
- [Evaluating the Robustness of Neural Language Models to Input Perturbations](https://mlsanity.com/paper/2108.12237/) (2021)
- [Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models](https://mlsanity.com/paper/2110.07736/) (2021)
- [Explore Spurious Correlations at the Concept Level in Language Models for Text Classification](https://mlsanity.com/paper/2311.08648/) (2023)
- [MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation](https://mlsanity.com/paper/2211.01355/) (2022)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval](https://mlsanity.com/paper/2212.10526/) (2022)
- [Interpreting the Robustness of Neural NLP Models to Textual Perturbations](https://mlsanity.com/paper/2110.07159/) (2021)
- [Efficient Inference for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2109.06679/) (2021)
- [On the Robustness of Intent Classification and Slot Labeling in Goal-oriented Dialog Systems to Real-world Noise](https://mlsanity.com/paper/2104.07149/) (2021)
- [An Empirical Study of In-context Learning in LLMs for Machine Translation](https://mlsanity.com/paper/2401.12097/) (2024)
- [MaxMatch-Dropout: Subword Regularization for WordPiece](https://mlsanity.com/paper/2209.04126/) (2022)
- [PGODE: Towards High-quality System Dynamics Modeling](https://mlsanity.com/paper/2311.06554/) (2023)
- [Hierarchical Interpretation of Neural Text Classification](https://mlsanity.com/paper/2202.09792/) (2022)
- [Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation](https://mlsanity.com/paper/2407.12832/) (2024)
- [On the data requirements of probing](https://mlsanity.com/paper/2202.12801/) (2022)
- [Generating Authentic Adversarial Examples beyond Meaning-preserving with Doubly Round-trip Translation](https://mlsanity.com/paper/2204.08689/) (2022)
- [Improving the Robustness of Summarization Models by Detecting and Removing Input Noise](https://mlsanity.com/paper/2212.09928/) (2022)
- [A Classification-Guided Approach for Adversarial Attacks against Neural Machine Translation](https://mlsanity.com/paper/2308.15246/) (2023)
- [Sometimes We Want Translationese](https://mlsanity.com/paper/2104.07623/) (2021)
- [Rare but Severe Neural Machine Translation Errors Induced by Minimal Deletion: An Empirical Study on Chinese and English](https://mlsanity.com/paper/2209.02145/) (2022)
- [Checks and Strategies for Enabling Code-Switched Machine Translation](https://mlsanity.com/paper/2210.05096/) (2022)
- [Adapting Large Language Models for Multi-Domain Retrieval-Augmented-Generation](https://mlsanity.com/paper/2504.02411/) (2025)
- [Faithful Target Attribute Prediction in Neural Machine Translation](https://mlsanity.com/paper/2109.12105/) (2021)
- [Challenges and Opportunities in Text Generation Explainability](https://mlsanity.com/paper/2405.08468/) (2024)
- [Understanding Pre-Editing for Black-Box Neural Machine Translation](https://mlsanity.com/paper/2102.02955/) (2021)
- [Doubly-Trained Adversarial Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2110.05691/) (2021)
- [Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition](https://mlsanity.com/paper/2401.10447/) (2024)
- [What Have We Achieved on Non-autoregressive Translation?](https://mlsanity.com/paper/2405.12788/) (2024)

## Abstract (arXiv)

Neural Machine Translation (NMT) models are sensitive to small perturbations in the input. Robustness to such perturbations is typically measured using translation quality metrics such as BLEU on the noisy input. This paper proposes additional metrics which measure the relative degradation and changes in translation when small perturbations are added to the input. We focus on a class of models employing subword regularization to address robustness and perform extensive evaluations of these models using the robustness measures proposed. Results show that our proposed metrics reveal a clear trend of improved robustness to perturbations when subword regularization methods are used.