# Software Vulnerability Prediction in Low-Resource Languages: An Empirical Study of CodeBERT and ChatGPT

arXiv: https://arxiv.org/abs/2404.17110 · HTML: https://mlsanity.com/paper/2404.17110/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 19 of 19)

- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [Vulnerability Detection with Fine-grained Interpretations](https://mlsanity.com/paper/2106.10478/) (2021)
- [Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges](https://mlsanity.com/paper/2002.05442/) (2020)
- [Prompt-Enhanced Software Vulnerability Detection Using ChatGPT](https://mlsanity.com/paper/2308.12697/) (2023)
- [An Empirical Study of Deep Learning Models for Vulnerability Detection](https://mlsanity.com/paper/2212.08109/) (2022)
- [ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We?](https://mlsanity.com/paper/2310.09810/) (2023)
- [LineVD: Statement-level Vulnerability Detection using Graph Neural Networks](https://mlsanity.com/paper/2203.05181/) (2022)
- [Evaluation of ChatGPT Model for Vulnerability Detection](https://mlsanity.com/paper/2304.07232/) (2023)
- [A Survey on Data-driven Software Vulnerability Assessment and Prioritization](https://mlsanity.com/paper/2107.08364/) (2021)
- [Automated Software Vulnerability Assessment with Concept Drift](https://mlsanity.com/paper/2103.11316/) (2021)
- [A Large-scale Study of Security Vulnerability Support on Developer Q&A Websites](https://mlsanity.com/paper/2008.04176/) (2020)
- [PUMiner: Mining Security Posts from Developer Question and Answer Websites with PU Learning](https://mlsanity.com/paper/2003.03741/) (2020)
- [Are Latent Vulnerabilities Hidden Gems for Software Vulnerability Prediction? An Empirical Study](https://mlsanity.com/paper/2401.11105/) (2024)
- [Automated Security Assessment for the Internet of Things](https://mlsanity.com/paper/2109.04029/) (2021)
- [On the Use of Fine-grained Vulnerable Code Statements for Software Vulnerability Assessment Models](https://mlsanity.com/paper/2203.08417/) (2022)
- [SoK: Machine Learning for Continuous Integration](https://mlsanity.com/paper/2304.02829/) (2023)
- [Systematic Literature Review on Application of Machine Learning in Continuous Integration](https://mlsanity.com/paper/2305.12695/) (2023)
- [Mitigating ML Model Decay in Continuous Integration with Data Drift Detection: An Empirical Study](https://mlsanity.com/paper/2305.12736/) (2023)

## Built on by (most cited first, 9 of 9)

- [LLMSecConfig: An LLM-Based Approach for Fixing Software Container Misconfigurations](https://mlsanity.com/paper/2502.02009/) (2025)
- [A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models](https://mlsanity.com/paper/2507.22659/) (2025)
- [Mitigating Data Imbalance for Software Vulnerability Assessment: Does Data Augmentation Help?](https://mlsanity.com/paper/2407.10722/) (2024)
- [Automated Code-centric Software Vulnerability Assessment: How Far Are We? An Empirical Study in C/C++](https://mlsanity.com/paper/2407.17053/) (2024)
- [Automatic Data Labeling for Software Vulnerability Prediction Models: How Far Are We?](https://mlsanity.com/paper/2407.17803/) (2024)
- [Securing the AI Supply Chain: What Can We Learn From Developer-Reported Security Issues and Solutions of AI Projects?](https://mlsanity.com/paper/2512.23385/) (2025)
- [VulTriage: Triple-Path Context Augmentation for LLM-Based Vulnerability Detection](https://mlsanity.com/paper/2605.09461/) (2026)
- [A systematic literature Review for Transformer-based Software Vulnerability detection](https://mlsanity.com/paper/2604.24822/) (2026)
- [Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attacks](https://mlsanity.com/paper/2609.09865/) (2026)

## Abstract (arXiv)

Background: Software Vulnerability (SV) prediction in emerging languages is increasingly important to ensure software security in modern systems. However, these languages usually have limited SV data for developing high-performing prediction models. Aims: We conduct an empirical study to evaluate the impact of SV data scarcity in emerging languages on the state-of-the-art SV prediction model and investigate potential solutions to enhance the performance. Method: We train and test the state-of-the-art model based on CodeBERT with and without data sampling techniques for function-level and line-level SV prediction in three low-resource languages - Kotlin, Swift, and Rust. We also assess the effectiveness of ChatGPT for low-resource SV prediction given its recent success in other domains. Results: Compared to the original work in C/C++ with large data, CodeBERT's performance of function-level and line-level SV prediction significantly declines in low-resource languages, signifying the negative impact of data scarcity. Regarding remediation, data sampling techniques fail to improve CodeBERT; whereas, ChatGPT showcases promising results, substantially enhancing predictive performance by up to 34.4% for the function level and up to 53.5% for the line level. Conclusion: We have highlighted the challenge and made the first promising step for low-resource SV prediction, paving the way for future research in this direction.