# Predicting Line-Level Defects by Capturing Code Contexts with Hierarchical Transformers

arXiv: https://arxiv.org/abs/2312.11889 · HTML: https://mlsanity.com/paper/2312.11889/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 59 references that are in the map.
- First posted on arXiv 19 Dec 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Recommendations for Datasets for Source Code Summarization](https://mlsanity.com/paper/1904.02660/) (2019)
- [CC2Vec: Distributed Representations of Code Changes](https://mlsanity.com/paper/2003.05620/) (2020)
- [On the Evaluation of Commit Message Generation Models: An Experimental Study](https://mlsanity.com/paper/2107.05373/) (2021)
- [JITLine: A Simpler, Better, Faster, Finer-grained Just-In-Time Defect Prediction](https://mlsanity.com/paper/2103.07068/) (2021)
- [Predicting Defective Lines Using a Model-Agnostic Technique](https://mlsanity.com/paper/2009.03612/) (2020)
- [Practitioners' Perceptions of the Goals and Visual Explanations of Defect Prediction Models](https://mlsanity.com/paper/2102.12007/) (2021)
- [Revisiting the Impact of Dependency Network Metrics on Software Defect Prediction](https://mlsanity.com/paper/2202.06145/) (2022)
- [Watch out for Extrinsic Bugs! A Case Study of their Impact in Just-In-Time Bug Prediction Models on the OpenStack project](https://mlsanity.com/paper/2103.15180/) (2021)
- [Defectors: A Large, Diverse Python Dataset for Defect Prediction](https://mlsanity.com/paper/2303.04738/) (2023)
- [Bugsplainer: Leveraging Code Structures to Explain Software Bugs with Neural Machine Translation](https://mlsanity.com/paper/2308.12267/) (2023)

## Abstract (arXiv)

Software defects consume 40% of the total budget in software development and cost the global economy billions of dollars every year. Unfortunately, despite the use of many software quality assurance (SQA) practices in software development (e.g., code review, continuous integration), defects may still exist in the official release of a software product. Therefore, prioritizing SQA efforts for the vulnerable areas of the codebase is essential to ensure the high quality of a software release. Predicting software defects at the line level could help prioritize the SQA effort but is a highly challenging task given that only ~3% of lines of a codebase could be defective. Existing works on line-level defect prediction often fall short and cannot fully leverage the line-level defect information. In this paper, we propose Bugsplorer, a novel deep-learning technique for line-level defect prediction. It leverages a hierarchical structure of transformer models to represent two types of code elements: code tokens and code lines. Unlike the existing techniques that are optimized for file-level defect prediction, Bugsplorer is optimized for a line-level defect prediction objective. Our evaluation with five performance metrics shows that Bugsplorer has a promising capability of predicting defective lines with 26-72% better accuracy than that of the state-of-the-art technique. It can rank the first 20% defective lines within the top 1-3% suspicious lines. Thus, Bugsplorer has the potential to significantly reduce SQA costs by ranking defective lines higher.