# ChatGPT's Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis Tools

arXiv: https://arxiv.org/abs/2409.06561 · HTML: https://mlsanity.com/paper/2409.06561/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 10 Sep 2024.

## Builds on (most cited first, 14 of 14)

- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions](https://mlsanity.com/paper/2308.02312/) (2023)
- [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://mlsanity.com/paper/2407.07064/) (2024)
- [CryptoGuard: High Precision Detection of Cryptographic Vulnerabilities in Massive-sized Java Projects](https://mlsanity.com/paper/1806.06881/) (2018)
- [The Impact of Developer Experience in Using Java Cryptography](https://mlsanity.com/paper/1908.01489/) (2019)
- [Java Cryptography Uses in the Wild](https://mlsanity.com/paper/2009.01101/) (2020)
- [FluentCrypto: Cryptography in Easy Mode](https://mlsanity.com/paper/2108.07211/) (2021)
- [Time to Separate from StackOverflow and Match with ChatGPT for Encryption](https://mlsanity.com/paper/2406.06164/) (2024)
- [Mutation-based Evaluation of Cryptographic API Misuse Detectors](https://mlsanity.com/paper/2107.07065/) (2021)
- [Cryptography Vulnerabilities on HackerOne](https://mlsanity.com/paper/2111.03859/) (2021)
- [CRYLOGGER: Detecting Crypto Misuses Dynamically](https://mlsanity.com/paper/2007.01061/) (2020)
- [Hurdles for Developers in Cryptography](https://mlsanity.com/paper/2108.07141/) (2021)
- [CamBench -- Cryptographic API Misuse Detection Tool Benchmark Suite](https://mlsanity.com/paper/2204.06447/) (2022)
- [From Struggle to Simplicity with a Usable and Secure API for Encryption in Java](https://mlsanity.com/paper/2409.05128/) (2024)

## Built on by (most cited first, 4 of 4)

- [Benchmarking Prompt Engineering Techniques for Secure Code Generation with GPT Models](https://mlsanity.com/paper/2502.06039/) (2025)
- [CIPHER: Cryptographic Insecurity Profiling via Hybrid Evaluation of Responses](https://mlsanity.com/paper/2602.01438/) (2026)
- [Beyond Static Tools: Evaluating Large Language Models for Cryptographic Misuse Detection](https://mlsanity.com/paper/2411.09772/) (2024)
- [Poisoned Source Code Detection in Code Models](https://mlsanity.com/paper/2502.13459/) (2025)

## Abstract (arXiv)

The correct adoption of cryptography APIs is challenging for mainstream developers, often resulting in widespread API misuse. Meanwhile, cryptography misuse detectors have demonstrated inconsistent performance and remain largely inaccessible to most developers. We investigated the extent to which ChatGPT can detect cryptography misuses and compared its performance with that of the state-of-the-art static analysis tools. Our investigation, mainly based on the CryptoAPI-Bench benchmark, demonstrated that ChatGPT is effective in identifying cryptography API misuses, and with the use of prompt engineering, it can even outperform leading static cryptography misuse detectors.