# CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices

arXiv: https://arxiv.org/abs/2609.21344 · HTML: https://mlsanity.com/paper/2609.21344/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 18 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://mlsanity.com/paper/2406.07599/) (2024)
- [SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity](https://mlsanity.com/paper/2412.20787/) (2024)
- [CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenges](https://mlsanity.com/paper/2504.19093/) (2025)
- [Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilities](https://mlsanity.com/paper/2505.24621/) (2025)
- [HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation](https://mlsanity.com/paper/2601.13864/) (2026)
- [Malware Detection at the Edge with Lightweight LLMs: A Performance Evaluation](https://mlsanity.com/paper/2503.04302/) (2025)
- [Evaluating Language Models For Threat Detection in IoT Security Logs](https://mlsanity.com/paper/2507.02390/) (2025)
- [AICrypto: Evaluating Cryptography Capabilities of Large Language Models](https://mlsanity.com/paper/2507.09580/) (2025)
- [Securing LLM-Generated Embedded Firmware through AI Agent-Driven Validation and Patching](https://mlsanity.com/paper/2509.09970/) (2025)
- [CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering](https://mlsanity.com/paper/2604.03750/) (2026)
- [HarmChip: Evaluating Hardware Security Centric LLM Safety via Jailbreak Benchmarking](https://mlsanity.com/paper/2604.17093/) (2026)
- [Large Language Model-driven Security Assistant for Internet of Things via Chain-of-Thought](https://mlsanity.com/paper/2505.06307/) (2025)
- [CrypFormBench: Benchmarking Formal Analysis Capability of Large Language Models for Cryptographic Schemes](https://mlsanity.com/paper/2606.25561/) (2026)
- [CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptography](https://mlsanity.com/paper/2512.02625/) (2025)
- [CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge](https://mlsanity.com/paper/2604.20389/) (2026)
- [CryptanalysisBench: Can LLMs do Cryptanalysis?](https://mlsanity.com/paper/2607.18538/) (2026)

## Abstract (arXiv)

For Internet of Things (IoT) devices, a secure algorithm alone is not enough: an attacker with physical access can attack the implementation directly, and its flaws are hard to fix once deployed. Large language models (LLMs) are now used to build and analyze such implementations. LLM benchmarks exist for cryptography and general cybersecurity, but none covers cryptographic engineering. In this paper, we present CESBench, 380 expert-written items across six sub-domains of cryptographic engineering security for IoT devices: side-channel, fault injection, implementation, countermeasures, evaluation, and integration. Four task types target different competences: 209 multiple-choice items test recall, 67 judgment items require a security verdict and its justification, 63 scenario items require an engineering diagnosis, and 41 code tasks are graded by 572 test cases. To validate the benchmark, 11 open-weight and proprietary LLMs answer every item. Multiple-choice and code responses are scored automatically, and judgment and scenario responses by an LLM judge, whose scores are checked against a second judge from another model family and human re-scoring. Composite scores range from 54.4% to 83.6%. The top score on each task type is 98.6% for multiple choice, 95.1% for code, and 88.4% for scenario diagnosis, but only 58.8% for judgment. Across models, 88.5% of verdicts are correct, yet their justifications earn only 53.4% of the rubric marks. Multiple choice is near its ceiling for the strongest models and most code tasks are solved, whereas justifying a security verdict remains the weakest competence. The benchmark, prompts, and per-item results are public.