# Can NMT Understand Me? Towards Perturbation-based Evaluation of NMT Models for Code Generation

arXiv: https://arxiv.org/abs/2203.15319 · HTML: https://mlsanity.com/paper/2203.15319/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 59 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow](https://mlsanity.com/paper/1805.08949/) (2018)
- [Counter-fitting Word Vectors to Linguistic Constraints](https://mlsanity.com/paper/1603.00892/) (2016)
- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://mlsanity.com/paper/1906.02443/) (2019)
- [NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System](https://mlsanity.com/paper/1802.08979/) (2018)
- [Neural Machine Translation with Extended Context](https://mlsanity.com/paper/1708.05943/) (2017)
- [Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples](https://mlsanity.com/paper/1803.01128/) (2018)
- [TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation](https://mlsanity.com/paper/1810.02720/) (2018)
- [On Adversarial Examples for Character-Level Neural Machine Translation](https://mlsanity.com/paper/1806.09030/) (2018)
- [Latent Predictor Networks for Code Generation](https://mlsanity.com/paper/1603.06744/) (2016)
- [TreeGen: A Tree-Based Transformer Architecture for Code Generation](https://mlsanity.com/paper/1911.09983/) (2019)
- [On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models](https://mlsanity.com/paper/1903.06620/) (2019)
- [Robust Text-to-SQL Generation with Execution-Guided Decoding](https://mlsanity.com/paper/1807.03100/) (2018)
- [Does BLEU Score Work for Code Migration?](https://mlsanity.com/paper/1906.04903/) (2019)
- [Exploiting Cross-Sentence Context for Neural Machine Translation](https://mlsanity.com/paper/1704.04347/) (2017)
- [Retrieval-Based Neural Code Generation](https://mlsanity.com/paper/1808.10025/) (2018)
- [Automatic Testing and Improvement of Machine Translation](https://mlsanity.com/paper/1910.02688/) (2019)
- [Incorporating External Knowledge through Pre-training for Natural Language to Code Generation](https://mlsanity.com/paper/2004.09015/) (2020)
- [TF-Coder: Program Synthesis for Tensor Manipulations](https://mlsanity.com/paper/2003.09040/) (2020)
- [How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?](https://mlsanity.com/paper/1704.04441/) (2017)
- [Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods](https://mlsanity.com/paper/2105.03311/) (2021)
- [Shellcode_IA32: A Dataset for Automatic Shellcode Generation](https://mlsanity.com/paper/2104.13100/) (2021)

## Built on by (most cited first, 2 of 2)

- [Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?](https://mlsanity.com/paper/2306.01220/) (2023)
- [Enhancing Robustness of AI Offensive Code Generators via Data Augmentation](https://mlsanity.com/paper/2306.05079/) (2023)

## Abstract (arXiv)

Neural Machine Translation (NMT) has reached a level of maturity to be recognized as the premier method for the translation between different languages and aroused interest in different research areas, including software engineering. A key step to validate the robustness of the NMT models consists in evaluating the performance of the models on adversarial inputs, i.e., inputs obtained from the original ones by adding small amounts of perturbation. However, when dealing with the specific task of the code generation (i.e., the generation of code starting from a description in natural language), it has not yet been defined an approach to validate the robustness of the NMT models. In this work, we address the problem by identifying a set of perturbations and metrics tailored for the robustness assessment of such models. We present a preliminary experimental evaluation, showing what type of perturbations affect the model the most and deriving useful insights for future directions.