Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detection
Not on the map. Why?
Abstract (arXiv)
Recent advances in defect detection use language models. Existing works enhanced the training data to improve the models' robustness when applied to semantically identical code (i.e., predictions should be the same). However, the use of semantically identical code has not been considered for improving the tools during their application - a concept closely related to metamorphic testing. The goal of our study is to determine whether we can use semantic-preserving transformations, analogue to mutation operators, to improve the performance of defect detection tools in the testing stage. We first collect existing publications which implemented semantic-preserving transformations and share their implementation, such that we can reuse them. We empirically study the effectiveness of three different ensemble strategies for enhancing defect detection tools. We apply the collected transformations on the Devign dataset, considering vulnerabilities as a type of defect, and two fine-tuned large language models for defect detection (VulBERTa, PLBART). We found 28 publications with 94 different transformations. We choose to implement 39 transformations from four of the publications, but a manual check revealed that 23 out 39 transformations change code semantics. Using the 16 remaining, correct transformations and three ensemble strategies, we were not able to increase the accuracy of the defect detection models. Our results show that reusing shared semantic-preserving transformation is difficult, sometimes even causing wrongful changes to the semantics. Keywords: defect detection, language model, semantic-preserving transformation, ensemble
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 37 of its 66 references that are in the map. First posted on arXiv 30 Mar 2025.
Builds on
37 of 66 references.
References · 37
- 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
- 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
- 2021Unified Pre-training for Program Understanding and Generationcited by 366
- 2022UniXcoder: Unified Cross-Modal Pre-training for Code Representationcited by 361
- 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
- 2018code2vec: Learning Distributed Representations of Codecited by 250
- 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 144
- 2020Contrastive Code Representation Learningcited by 76
- 2022Natural Attack for Pre-trained Models of Codecited by 75
- 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
- 2019Adversarial Examples for Models of Codecited by 63
- 2021CoTexT: Multi-task Learning with Code-Text Transformercited by 61
25 more
- 2021Self-Supervised Bug Detection and Repaircited by 59
- 2022NatGen: Generative pre-training by "Naturalizing" source codecited by 54
- 2022VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detectioncited by 51
- 2020Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformationscited by 49
- 2023Stealthy Backdoor Attack for Code Modelscited by 49
- 2021A Survey on Ensemble Learning under the Era of Deep Learningcited by 47
- 2020Semantic Robustness of Models of Source Codecited by 45
- 2020On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformationscited by 44
- 2020Adversarial Robustness for Codecited by 38
- 2019Misleading Authorship Attribution of Source Code using Adversarial Learningcited by 33
- 2020Exploring Software Naturalness through Neural Language Modelscited by 32
- 2023Uncovering the Limits of Machine Learning for Automatic Vulnerability Detectioncited by 30
- 2021Generating Adversarial Computer Programs using Optimized Obfuscationscited by 22
- 2021VELVET: a noVel Ensemble Learning approach to automatically locate VulnErable sTatementscited by 20
- 2023ContraBERT: Enhancing Code Pre-trained Models via Contrastive Learningcited by 16
- 2022RoPGen: Towards Robust Code Authorship Attribution via Automatic Coding Style Transformationcited by 15
- 2021A Search-Based Testing Framework for Deep Neural Networks of Source Code Embeddingcited by 13
- 2021On using distributed representations of source code for the detection of C security vulnerabilitiescited by 10
- 2022Improving Automated Program Repair with Domain Adaptationcited by 10
- 2020STRATA: Simple, Gradient-Free Attacks for Models of Codecited by 5
- 2023Boosting Source Code Learning with Text-Oriented Data Augmentation: An Empirical Studycited by 5
- 2019A Machine-learning Based Ensemble Method For Anti-patterns Detectioncited by 4
- 2021CoCoFuzzing: Testing Neural Code Models with Coverage-Guided Fuzzingcited by 4
- 2022MIXCODE: Enhancing Code Classification by Mixup-Based Data Augmentationcited by 4
- 2024Challenging Machine Learning Algorithms in Predicting Vulnerable JavaScript Functionscited by 1
Built on by
Citing papers per year: 2025: 2.
2 papers.