# CATBERT: Context-Aware Tiny BERT for Detecting Social Engineering Emails

arXiv: https://arxiv.org/abs/2010.03484 · HTML: https://mlsanity.com/paper/2010.03484/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 7 of 7)

- [CySecBERT: A Domain-Adapted Language Model for the Cybersecurity Domain](https://mlsanity.com/paper/2212.02974/) (2022)
- [FedAdapter: Efficient Federated Learning for Modern NLP](https://mlsanity.com/paper/2205.10162/) (2022)
- [Universal Spam Detection using Transfer Learning of BERT Model](https://mlsanity.com/paper/2202.03480/) (2022)
- [An Explainable Transformer-based Model for Phishing Email Detection: A Large Language Model Approach](https://mlsanity.com/paper/2402.13871/) (2024)
- [Using BERT Encoding to Tackle the Mad-lib Attack in SMS Spam Detection](https://mlsanity.com/paper/2107.06400/) (2021)
- [A Modular and Adaptive System for Business Email Compromise Detection](https://mlsanity.com/paper/2308.10776/) (2023)
- [AURA: Adaptive Uncertainty-Routed Analysis for Email Threat Detection](https://mlsanity.com/paper/2609.19873/) (2026)

## Abstract (arXiv)

Targeted phishing emails are on the rise and facilitate the theft of billions of dollars from organizations a year. While malicious signals from attached files or malicious URLs in emails can be detected by conventional malware signatures or machine learning technologies, it is challenging to identify hand-crafted social engineering emails which don't contain any malicious code and don't share word choices with known attacks. To tackle this problem, we fine-tune a pre-trained BERT model by replacing the half of Transformer blocks with simple adapters to efficiently learn sophisticated representations of the syntax and semantics of the natural language. Our Context-Aware network also learns the context representations between email's content and context features from email headers. Our CatBERT(Context-Aware Tiny Bert) achieves a 87% detection rate as compared to DistilBERT, LSTM, and logistic regression baselines which achieve 83%, 79%, and 54% detection rates at false positive rates of 1%, respectively. Our model is also faster than competing transformer approaches and is resilient to adversarial attacks which deliberately replace keywords with typos or synonyms.