# Universal Spam Detection using Transfer Learning of BERT Model

arXiv: https://arxiv.org/abs/2202.03480 · HTML: https://mlsanity.com/paper/2202.03480/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 68 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [K-BERT: Enabling Language Representation with Knowledge Graph](https://mlsanity.com/paper/1909.07606/) (2019)
- [Activation Functions: Comparison of trends in Practice and Research for Deep Learning](https://mlsanity.com/paper/1811.03378/) (2018)
- [On Loss Functions for Deep Neural Networks in Classification](https://mlsanity.com/paper/1702.05659/) (2017)
- [An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family](https://mlsanity.com/paper/1511.05042/) (2015)
- [CATBERT: Context-Aware Tiny BERT for Detecting Social Engineering Emails](https://mlsanity.com/paper/2010.03484/) (2020)
- [Can Active Memory Replace Attention?](https://mlsanity.com/paper/1610.08613/) (2016)
- [Classification of Spam Emails through Hierarchical Clustering and Supervised Learning](https://mlsanity.com/paper/2005.08773/) (2020)

## Built on by (most cited first, 13 of 13)

- [Large Language Models for Education: A Survey](https://mlsanity.com/paper/2405.13001/) (2024)
- [Spam-T5: Benchmarking Large Language Models for Few-Shot Email Spam Detection](https://mlsanity.com/paper/2304.01238/) (2023)
- [An Improved Transformer-based Model for Detecting Phishing, Spam, and Ham: A Large Language Model Approach](https://mlsanity.com/paper/2311.04913/) (2023)
- [Spam Detection Using BERT](https://mlsanity.com/paper/2206.02443/) (2022)
- [Credit Risk Meets Large Language Models: Building a Risk Indicator from Loan Descriptions in P2P Lending](https://mlsanity.com/paper/2401.16458/) (2024)
- [Email Spam Detection Using Hierarchical Attention Hybrid Deep Learning Method](https://mlsanity.com/paper/2204.07390/) (2022)
- [Privacy-Preserving Deep Learning Model for Covid-19 Disease Detection](https://mlsanity.com/paper/2209.04445/) (2022)
- [ExplainableDetector: Exploring Transformer-based Language Modeling Approach for SMS Spam Detection with Explainability Analysis](https://mlsanity.com/paper/2405.08026/) (2024)
- [Investigating Pre-trained Language Models on Cross-Domain Datasets, a Step Closer to General AI](https://mlsanity.com/paper/2306.12205/) (2023)
- [A Novel Dataset for Financial Education Text Simplification in Spanish](https://mlsanity.com/paper/2312.09897/) (2023)
- [ScamSpot: Fighting Financial Fraud in Instagram Comments](https://mlsanity.com/paper/2402.08869/) (2024)
- [Effects of Soft-Domain Transfer and Named Entity Information on Deception Detection](https://mlsanity.com/paper/2410.14814/) (2024)
- [Unified Kernel-Segregated Transpose Convolution Operation](https://mlsanity.com/paper/2502.20493/) (2025)

## Abstract (arXiv)

Deep learning transformer models become important by training on text data based on self-attention mechanisms. This manuscript demonstrated a novel universal spam detection model using pre-trained Google's Bidirectional Encoder Representations from Transformers (BERT) base uncased models with four datasets by efficiently classifying ham or spam emails in real-time scenarios. Different methods for Enron, Spamassain, Lingspam, and Spamtext message classification datasets, were used to train models individually in which a single model was obtained with acceptable performance on four datasets. The Universal Spam Detection Model (USDM) was trained with four datasets and leveraged hyperparameters from each model. The combined model was finetuned with the same hyperparameters from these four models separately. When each model using its corresponding dataset, an F1-score is at and above 0.9 in individual models. An overall accuracy reached 97%, with an F1 score of 0.96. Research results and implications were discussed.