# CIPHER: Cryptographic Insecurity Profiling via Hybrid Evaluation of Responses

arXiv: https://arxiv.org/abs/2602.01438 · HTML: https://mlsanity.com/paper/2602.01438/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 46 references that are in the map.
- First posted on arXiv 1 Feb 2026.

## Builds on (most cited first, 5 of 5)

- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities](https://mlsanity.com/paper/2509.03331/) (2025)
- [Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairs](https://mlsanity.com/paper/2409.00571/) (2024)
- [ChatGPT's Potential in Cryptography Misuse Detection: A Comparative Analysis with Static Analysis Tools](https://mlsanity.com/paper/2409.06561/) (2024)

## Built on by (most cited first, 1 of 1)

- [CIBER: A Comprehensive Benchmark for Security Evaluation of Code Interpreter Agents](https://mlsanity.com/paper/2602.19547/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used to assist developers with code, yet their implementations of cryptographic functionality often contain exploitable flaws. Minor design choices (e.g., static initialization vectors or missing authentication) can silently invalidate security guarantees. We introduce CIPHER(Cryptographic Insecurity Profiling via Hybrid Evaluation of Responses), a benchmark for measuring cryptographic vulnerability incidence in LLM-generated Python code under controlled security-guidance conditions. CIPHER uses insecure/neutral/secure prompt variants per task, a cryptography-specific vulnerability taxonomy, and line-level attribution via an automated scoring pipeline. Across a diverse set of widely used LLMs, we find that explicit secure prompting reduces some targeted issues but does not reliably eliminate cryptographic vulnerabilities overall. The benchmark and reproducible scoring pipeline will be publicly released upon publication.