# CLAWSAT: Towards Both Robust and Accurate Code Models

arXiv: https://arxiv.org/abs/2211.11711 · HTML: https://mlsanity.com/paper/2211.11711/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 54 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 30 of 40)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017)
- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [Robust Pre-Training by Adversarial Contrastive Learning](https://mlsanity.com/paper/2010.13337/) (2020)
- [Adversarial Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2006.07589/) (2020)
- [Is Robustness the Cost of Accuracy? -- A Comprehensive Study on the Robustness of 18 Deep Image Classification Models](https://mlsanity.com/paper/1808.01688/) (2018)
- [You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion](https://mlsanity.com/paper/2007.02220/) (2020)
- [Jigsaw: Large Language Models meet Program Synthesis](https://mlsanity.com/paper/2112.02969/) (2021)
- [A Convolutional Attention Network for Extreme Summarization of Source Code](https://mlsanity.com/paper/1602.03001/) (2016)
- [When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?](https://mlsanity.com/paper/2111.01124/) (2021)
- [Contrastive Code Representation Learning](https://mlsanity.com/paper/2007.04973/) (2020)
- [Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimization](https://mlsanity.com/paper/2112.12376/) (2021)
- [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022)
- [Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Tree](https://mlsanity.com/paper/2002.08653/) (2020)
- [Adversarial Examples for Models of Code](https://mlsanity.com/paper/1910.07517/) (2019)
- [SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation](https://mlsanity.com/paper/2108.04556/) (2021)
- [Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations](https://mlsanity.com/paper/2009.02731/) (2020)
- [Semantic Robustness of Models of Source Code](https://mlsanity.com/paper/2002.03043/) (2020)

## Built on by (most cited first, 2 of 2)

- [Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code](https://mlsanity.com/paper/2312.14856/) (2023)
- [A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair](https://mlsanity.com/paper/2604.21579/) (2026)

## Abstract (arXiv)

We integrate contrastive learning (CL) with adversarial learning to co-optimize the robustness and accuracy of code models. Different from existing works, we show that code obfuscation, a standard code transformation operation, provides novel means to generate complementary `views' of a code that enable us to achieve both robust and accurate code models. To the best of our knowledge, this is the first systematic study to explore and exploit the robustness and accuracy benefits of (multi-view) code obfuscations in code models. Specifically, we first adopt adversarial codes as robustness-promoting views in CL at the self-supervised pre-training phase. This yields improved robustness and transferability for downstream tasks. Next, at the supervised fine-tuning stage, we show that adversarial training with a proper temporally-staggered schedule of adversarial code generation can further improve robustness and accuracy of the pre-trained code model. Built on the above two modules, we develop CLAWSAT, a novel self-supervised learning (SSL) framework for code by integrating underlinetextrmCL with underlinetextrmadversarial vieunderlinetextrmws (CLAW) with underlinetextrmstaggered underlinetextrmadversarial underlinetextrmtraining (SAT). On evaluating three downstream tasks across Python and Java, we show that CLAWSAT consistently yields the best robustness and accuracy (textite.g. 11% in robustness and 6% in accuracy on the code summarization task in Python). We additionally demonstrate the effectiveness of adversarial learning in CLAW by analyzing the characteristics of the loss landscape and interpretability of the pre-trained models.