# Beyond Static Tools: Evaluating Large Language Models for Cryptographic Misuse Detection

arXiv: https://arxiv.org/abs/2411.09772 · HTML: https://mlsanity.com/paper/2411.09772/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 14 Nov 2024.

## Builds on (most cited first, 9 of 9)

- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Large Language Models for Code Analysis: Do LLMs Really Do Their Job?](https://mlsanity.com/paper/2310.12357/) (2023)
- [Exploring ChatGPT's Capabilities on Vulnerability Management](https://mlsanity.com/paper/2311.06530/) (2023)
- [ChatGPT, Can You Generate Solutions for my Coding Exercises? An Evaluation on its Effectiveness in an undergraduate Java Programming Course](https://mlsanity.com/paper/2305.13680/) (2023)
- [CryptoGuard: High Precision Detection of Cryptographic Vulnerabilities in Massive-sized Java Projects](https://mlsanity.com/paper/1806.06881/) (2018)
- [Beyond Static Pattern Matching? Rethinking Automatic Cryptographic API Misuse Detection in the Era of LLMs](https://mlsanity.com/paper/2407.16576/) (2024)
- [ChatGPT's Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis Tools](https://mlsanity.com/paper/2409.06561/) (2024)
- [FluentCrypto: Cryptography in Easy Mode](https://mlsanity.com/paper/2108.07211/) (2021)
- [Mutation-based Evaluation of Cryptographic API Misuse Detectors](https://mlsanity.com/paper/2107.07065/) (2021)

## Abstract (arXiv)

The use of Large Language Models (LLMs) in software development is rapidly growing, with developers increasingly relying on these models for coding assistance, including security-critical tasks. Our work presents a comprehensive comparison between traditional static analysis tools for cryptographic API misuse detection-CryptoGuard, CogniCrypt, and Snyk Code-and the LLMs-GPT and Gemini. Using benchmark datasets (OWASP, CryptoAPI, and MASC), we evaluate the effectiveness of each tool in identifying cryptographic misuses. Our findings show that GPT 4-o-mini surpasses current state-of-the-art static analysis tools on the CryptoAPI and MASC datasets, though it lags on the OWASP dataset. Additionally, we assess the quality of LLM responses to determine which models provide actionable and accurate advice, giving developers insights into their practical utility for secure coding. This study highlights the comparative strengths and limitations of static analysis versus LLM-driven approaches, offering valuable insights into the evolving role of AI in advancing software security practices.