# An Empirical Study on the Effectiveness of Large Language Models for SATD Identification and Classification

arXiv: https://arxiv.org/abs/2405.06806 · HTML: https://mlsanity.com/paper/2405.06806/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2024.

## Built on by (most cited first, 4 of 4)

- [SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation](https://mlsanity.com/paper/2507.09108/) (2025)
- [Descriptor: C++ Self-Admitted Technical Debt Dataset (CppSATD)](https://mlsanity.com/paper/2505.01136/) (2025)
- [A systematic review of generative AI usage for IT project management](https://mlsanity.com/paper/2604.21958/) (2026)
- [Watts and Debts of Agentic Frameworks: An Empirical Study (Registered Report)](https://mlsanity.com/paper/2606.10702/) (2026)

## Abstract (arXiv)

Self-Admitted Technical Debt (SATD), a concept highlighting sub-optimal choices in software development documented in code comments or other project resources, poses challenges in the maintainability and evolution of software systems. Large language models (LLMs) have demonstrated significant effectiveness across a broad range of software tasks, especially in software text generation tasks. Nonetheless, their effectiveness in tasks related to SATD is still under-researched. In this paper, we investigate the efficacy of LLMs in both identification and classification of SATD. For both tasks, we investigate the performance gain from using more recent LLMs, specifically the Flan-T5 family, across different common usage settings. Our results demonstrate that for SATD identification, all fine-tuned LLMs outperform the best existing non-LLM baseline, i.e., the CNN model, with a 4.4% to 7.2% improvement in F1 score. In the SATD classification task, while our largest fine-tuned model, Flan-T5-XL, still led in performance, the CNN model exhibited competitive results, even surpassing four of six LLMs. We also found that the largest Flan-T5 model, i.e., Flan-T5-XXL, when used with a zero-shot in-context learning (ICL) approach for SATD identification, provides competitive results with traditional approaches but performs 6.4% to 9.2% worse than fine-tuned LLMs. For SATD classification, few-shot ICL approach, incorporating examples and category descriptions in prompts, outperforms the zero-shot approach and even surpasses the fine-tuned smaller Flan-T5 models. Moreover, our experiments demonstrate that incorporating contextual information, such as surrounding code, into the SATD classification task enables larger fine-tuned LLMs to improve their performance.