# Trustworthiness Calibration Framework for Phishing Email Detection Using Large Language Models

arXiv: https://arxiv.org/abs/2511.04728 · HTML: https://mlsanity.com/paper/2511.04728/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 10 references that are in the map.
- First posted on arXiv 6 Nov 2025.

## Builds on (most cited first, 7 of 7)

- [Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects](https://mlsanity.com/paper/2412.00586/) (2024)
- [Forewarned is Forearmed: A Survey on Large Language Model-based Agents in Autonomous Cyberattacks](https://mlsanity.com/paper/2505.12786/) (2025)
- [An Improved Transformer-based Model for Detecting Phishing, Spam, and Ham: A Large Language Model Approach](https://mlsanity.com/paper/2311.04913/) (2023)
- [Debate-Driven Multi-Agent LLMs for Phishing Email Detection](https://mlsanity.com/paper/2503.22038/) (2025)
- [Enhancing Phishing Email Identification with Large Language Models](https://mlsanity.com/paper/2502.04759/) (2025)
- [SpearBot: Leveraging Large Language Models in a Generative-Critique Framework for Spear-Phishing Email Generation](https://mlsanity.com/paper/2412.11109/) (2024)
- [Evaluating the Efficacy of Large Language Models in Identifying Phishing Attempts](https://mlsanity.com/paper/2404.15485/) (2024)

## Abstract (arXiv)

Phishing emails continue to pose a persistent challenge to online communication, exploiting human trust and evading automated filters through realistic language and adaptive tactics. While large language models (LLMs) such as GPT-4 and LLaMA-3-8B achieve strong accuracy in text classification, their deployment in security systems requires assessing reliability beyond benchmark performance. To address this, this study introduces the Trustworthiness Calibration Framework (TCF), a reproducible methodology for evaluating phishing detectors across three dimensions: calibration, consistency, and robustness. These components are integrated into a bounded index, the Trustworthiness Calibration Index (TCI), and complemented by the Cross-Dataset Stability (CDS) metric that quantifies stability of trustworthiness across datasets. Experiments conducted on five corpora, such as SecureMail 2025, Phishing Validation 2024, CSDMC2010, Enron-Spam, and Nazario, using DeBERTa-v3-base, LLaMA-3-8B, and GPT-4 demonstrate that GPT-4 achieves the strongest overall trust profile, followed by LLaMA-3-8B and DeBERTa-v3-base. Statistical analysis confirms that reliability varies independently of raw accuracy, underscoring the importance of trust-aware evaluation for real-world deployment. The proposed framework establishes a transparent and reproducible foundation for assessing model dependability in LLM-based phishing detection.