# When Intelligence Fails: An Empirical Study on Why LLMs Struggle with Password Cracking

arXiv: https://arxiv.org/abs/2510.17884 · HTML: https://mlsanity.com/paper/2510.17884/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 21 references that are in the map.
- First posted on arXiv 18 Oct 2025.

## Builds on (most cited first, 8 of 8)

- [PassGAN: A Deep Learning Approach for Password Guessing](https://mlsanity.com/paper/1709.00440/) (2017)
- [PassFlow: Guessing Passwords with Generative Flows](https://mlsanity.com/paper/2105.06165/) (2021)
- [PassGPT: Password Modeling and (Guided) Generation with Large Language Models](https://mlsanity.com/paper/2306.01545/) (2023)
- [SE#PCFG: Semantically Enhanced PCFG for Password Analysis and Cracking](https://mlsanity.com/paper/2306.06824/) (2023)
- [Universal Neural-Cracking-Machines: Self-Configurable Password Models from Auxiliary Data](https://mlsanity.com/paper/2301.07628/) (2023)
- [On Deep Learning in Password Guessing, a Survey](https://mlsanity.com/paper/2208.10413/) (2022)
- [Search-based Ordered Password Generation of Autoregressive Neural Networks](https://mlsanity.com/paper/2403.09954/) (2024)
- [PassTSL: Modeling Human-Created Passwords through Two-Stage Learning](https://mlsanity.com/paper/2407.14145/) (2024)

## Abstract (arXiv)

The remarkable capabilities of Large Language Models (LLMs) in natural language understanding and generation have sparked interest in their potential for cybersecurity applications, including password guessing. In this study, we conduct an empirical investigation into the efficacy of pre-trained LLMs for password cracking using synthetic user profiles. Specifically, we evaluate the performance of state-of-the-art open-source LLMs such as TinyLLaMA, Falcon-RW-1B, and Flan-T5 by prompting them to generate plausible passwords based on structured user attributes (e.g., name, birthdate, hobbies). Our results, measured using Hit@1, Hit@5, and Hit@10 metrics under both plaintext and SHA-256 hash comparisons, reveal consistently poor performance, with all models achieving less than 1.5% accuracy at Hit@10. In contrast, traditional rule-based and combinator-based cracking methods demonstrate significantly higher success rates. Through detailed analysis and visualization, we identify key limitations in the generative reasoning of LLMs when applied to the domain-specific task of password guessing. Our findings suggest that, despite their linguistic prowess, current LLMs lack the domain adaptation and memorization capabilities required for effective password inference, especially in the absence of supervised fine-tuning on leaked password datasets. This study provides critical insights into the limitations of LLMs in adversarial contexts and lays the groundwork for future efforts in secure, privacy-preserving, and robust password modeling.