# On Adversarial Robustness of Synthetic Code Generation

arXiv: https://arxiv.org/abs/2106.11629 · HTML: https://mlsanity.com/paper/2106.11629/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 22 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Adversarial Examples: Attacks and Defenses for Deep Learning](https://mlsanity.com/paper/1712.07107/) (2017)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [DeepCoder: Learning to Write Programs](https://mlsanity.com/paper/1611.01989/) (2016)
- [RobustFill: Neural Program Learning under Noisy I/O](https://mlsanity.com/paper/1703.07469/) (2017)
- [Conditional BERT Contextual Augmentation](https://mlsanity.com/paper/1812.06705/) (2018)
- [Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis](https://mlsanity.com/paper/1805.04276/) (2018)
- [Massive Exploration of Neural Machine Translation Architectures](https://mlsanity.com/paper/1703.03906/) (2017)
- [Learning to Infer Program Sketches](https://mlsanity.com/paper/1902.06349/) (2019)
- [NAPS: Natural Program Synthesis Dataset](https://mlsanity.com/paper/1807.03168/) (2018)
- [Neural Program Search: Solving Programming Tasks from Description and Examples](https://mlsanity.com/paper/1802.04335/) (2018)
- [The First Shared Task on Discourse Representation Structure Parsing](https://mlsanity.com/paper/2005.13399/) (2020)
- [Ain't Nobody Got Time For Coding: Structure-Aware Program Synthesis From Natural Language](https://mlsanity.com/paper/1810.09717/) (2018)

## Built on by (most cited first, 3 of 3)

- [Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code](https://mlsanity.com/paper/2312.14856/) (2023)
- [CodeFort: Robust Training for Code Generation Models](https://mlsanity.com/paper/2405.01567/) (2024)
- [Multi-view Representation Learning from Malware to Defend Against Adversarial Variants](https://mlsanity.com/paper/2210.15429/) (2022)

## Abstract (arXiv)

Automatic code synthesis from natural language descriptions is a challenging task. We witness massive progress in developing code generation systems for domain-specific languages (DSLs) employing sequence-to-sequence deep learning techniques in the recent past. In this paper, we specifically experiment with textscAlgoLisp DSL-based generative models and showcase the existence of significant dataset bias through different classes of adversarial examples. We also experiment with two variants of Transformer-based models that outperform all existing textscAlgoLisp DSL-based code generation baselines. Consistent with the current state-of-the-art systems, our proposed models, too, achieve poor performance under adversarial settings. Therefore, we propose several dataset augmentation techniques to reduce bias and showcase their efficacy using robust experimentation.