mlsanityData to 7 Oct 2026

Automatic Testing and Improvement of Machine Translation

arXiv 1910.02688 ↗First posted 7 Oct 2019cs.SEcited by 35

Not on the map. Why?

Abstract (arXiv)

This paper presents TransRepair, a fully automatic approach for testing and repairing the consistency of machine translation systems. TransRepair combines mutation with metamorphic testing to detect inconsistency bugs (without access to human oracles). It then adopts probability-reference or cross-reference to post-process the translations, in a grey-box or black-box manner, to repair the inconsistencies. Our evaluation on two state-of-the-art translators, Google Translate and Transformer, indicates that TransRepair has a high precision (99%) on generating input pairs with consistent translations. With these tests, using automatic consistency metrics and manual assessment, we find that Google Translate and Transformer have approximately 36% and 40% inconsistency bugs. Black-box repair fixes 28% and 19% bugs on average for Google Translate and Transformer. Grey-box repair fixes 30% bugs on average for Transformer. Manual inspection indicates that the translations repaired by our approach improve consistency in 87% of cases (degrading it in 2%), and that our repairs have better translation acceptability in 27% of the cases (worse in 8%).

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 5; 2023: 14; 2024: 7; 2025: 4; 2026: 1.

35 papers.

Sort:

By area:

  1. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
  2. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulationcited by 43
  3. 2022Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Modelscited by 28
  4. 2025Hallucination Detection in Large Language Models with Metamorphic Relationscited by 24
  5. 2022SALTED: A Framework for SAlient Long-Tail Translation Error Detectioncited by 23
  6. 2024Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputscited by 22
  7. 2023(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIscited by 21
  8. 2023MTTM: Metamorphic Testing for Textual Content Moderation Softwarecited by 20
  9. 2023To Be Forgotten or To Be Fair: Unveiling Fairness Implications of Machine Unlearning Methodscited by 18
  10. 2022LLMEffiChecker: Understanding and Testing Efficiency Degradation of Large Language Modelscited by 16
  11. 2022AEON: A Method for Automatic Evaluation of NLP Test Casescited by 15
  12. 2023Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinkingcited by 14
23 more
  1. 2021Towards Practical Robustness Analysis for DNNs based on PAC-Model Learningcited by 12
  2. 2023Preventing Discriminatory Decision-making in Evolving Data Streamscited by 12
  3. 2024SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezingcited by 12
  4. 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
  5. 2021MDPFuzz: Testing Models Solving Markov Decision Processescited by 11
  6. 2021Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning Modelscited by 6
  7. 2023VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Modelscited by 6
  8. 2024A Large-Scale Empirical Study on Improving the Fairness of Image Classification Modelscited by 6
  9. 2023FairPy: A Toolkit for Evaluation of Prediction Biases and their Mitigation in Large Language Modelscited by 5
  10. 2023Validating Multimedia Content Moderation Software via Semantic Fusioncited by 5
  11. 2023Automated Testing and Improvement of Named Entity Recognition Systemscited by 3
  12. 2025HInter: Exposing Hidden Intersectional Bias in Large Language Modelscited by 3
  13. 2022Can NMT Understand Me? Towards Perturbation-based Evaluation of NMT Models for Code Generationcited by 2
  14. 2023Retromorphic Testing: A New Approach to the Test Oracle Problemcited by 2
  15. 2024Machine Translation Testing via Syntactic Tree Pruningcited by 2
  16. 2021Energy-bounded Learning for Robust Models of Codecited by 1
  17. 2023Towards General Error Diagnosis via Behavioral Testing in Machine Translationcited by 1
  18. 2023DREAM: Debugging and Repairing AutoML Pipelinescited by 1
  19. 2023Distribution-aware Fairness Test Generationcited by 0
  20. 2024Adversarial Testing for Visual Grounding via Image-Aware Property Reductioncited by 0
  21. 2024DeepKnowledge: Generalisation-Driven Deep Learning Testingcited by 0
  22. 2025TigAug: Data Augmentation for Testing Traffic Light Detection in Autonomous Driving Systemscited by 0
  23. 2026The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakescited by 0