mlsanityData to 7 Oct 2026

On Distribution Shift in Learning-based Bug Detectors

arXiv 2204.10049 ↗First posted 21 Apr 2022cs.LG cs.PL cs.SEcited by 15

Not on the map. Why?

Abstract (arXiv)

Deep learning has recently achieved initial success in program analysis tasks such as bug detection. Lacking real bugs, most existing works construct training and test data by injecting synthetic bugs into correct programs. Despite achieving high test accuracy (e.g., 90%), the resulting bug detectors are found to be surprisingly unusable in practice, i.e., <10% precision when used to scan real software repositories. In this work, we argue that this massive performance difference is caused by a distribution shift, i.e., a fundamental mismatch between the real bug distribution and the synthetic bug distribution used to train and evaluate the detectors. To address this key challenge, we propose to train a bug detector in two phases, first on a synthetic bug distribution to adapt the model to the bug detection domain, and then on a real bug distribution to drive the model towards the real distribution. During these two phases, we leverage a multi-task hierarchy, focal loss, and contrastive learning to further boost performance. We evaluate our approach extensively on three widely studied bug types, for which we construct new datasets carefully designed to capture the real bug distribution. The results demonstrate that our approach is practically effective and successfully mitigates the distribution shift: our learned detectors are highly performant on both our test set and the latest version of open source repositories. Our code, datasets, and models are publicly available at https://github.com/eth-sri/learning-real-bug-detector.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 21 Apr 2022.

Builds on

28 of 50 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  6. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  7. 2017A Survey on Multi-Task Learningcited by 838
  8. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  9. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  10. 2018code2vec: Learning Distributed Representations of Codecited by 249
  11. 2017Learning to Represent Programs with Graphscited by 219
  12. 2019Dice Loss for Data-imbalanced NLP Taskscited by 180
16 more
  1. 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 123
  2. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 90
  3. 2020Contrastive Code Representation Learningcited by 76
  4. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  5. 2018DeepBugs: A Learning Approach to Name-based Bug Detectioncited by 65
  6. 2021Language-Agnostic Representation Learning of Source Code from Structure and Contextcited by 63
  7. 2018Generative Code Modeling with Graphscited by 59
  8. 2021Self-Supervised Bug Detection and Repaircited by 59
  9. 2020LambdaNet: Probabilistic Type Inference using Graph Neural Networkscited by 44
  10. 2019Neural Program Repair by Jointly Learning to Localize and Repaircited by 39
  11. 2020Adversarial Robustness for Codecited by 36
  12. 2021Neural Transfer Learning for Repairing Security Vulnerabilities in C Codecited by 36
  13. 2019How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Datasetcited by 35
  14. 2018Learning to Represent Editscited by 34
  15. 2020Typilus: Neural Type Hintscited by 29
  16. 2021Neural Program Generation Modulo Static Analysiscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 2; 2025: 1; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  2. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  3. 2023Uncovering the Limits of Machine Learning for Automatic Vulnerability Detectioncited by 30
  4. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  5. 2023Large Language Models of Code Fail at Completing Code with Potential Bugscited by 28
  6. 2022Improving Automated Program Repair with Domain Adaptationcited by 10
  7. 2023RunBugRun -- An Executable Dataset for Automated Program Repaircited by 10
  8. 2022Detect-Localize-Repair: A Unified Framework for Learning to Debug with CodeT5cited by 9
  9. 2024DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Modelscited by 7
  10. 2023Learning Defect Prediction from Unrealistic Datacited by 5
  11. 2022Can we learn from developer mistakes? Learning to localize and repair real bugs from real bug fixescited by 4
  12. 2025Multilingual Multimodal Software Developer for Code Generationcited by 2
3 more
  1. 2024The Power of Types: Exploring the Impact of Type Checking on Neural Bug Detection in Dynamically Typed Languagescited by 1
  2. 2026Anchored Self-Play for Code Repaircited by 1
  3. 2026On the Robustness of LLMs' Internal Representation of Code Correctnesscited by 1