# Predicting Issue Types with seBERT

arXiv: https://arxiv.org/abs/2205.01335 · HTML: https://mlsanity.com/paper/2205.01335/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [On the validity of pre-trained transformers for natural language processing in the software engineering domain](https://mlsanity.com/paper/2109.04738/) (2021)
- [Predicting Issue Types on GitHub](https://mlsanity.com/paper/2107.09936/) (2021)
- [On the feasibility of automated prediction of bug and non-bug issues](https://mlsanity.com/paper/2003.05357/) (2020)

## Built on by (most cited first, 2 of 2)

- [Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP](https://mlsanity.com/paper/2209.07623/) (2022)
- [Combating Toxic Language: A Review of LLM-Based Strategies for Software Engineering](https://mlsanity.com/paper/2504.15439/) (2025)

## Abstract (arXiv)

Pre-trained transformer models are the current state-of-the-art for natural language models processing. seBERT is such a model, that was developed based on the BERT architecture, but trained from scratch with software engineering data. We fine-tuned this model for the NLBSE challenge for the task of issue type prediction. Our model dominates the baseline fastText for all three issue types in both recall and precisio} to achieve an overall F1-score of 85.7%, which is an increase of 4.1% over the baseline.