# Continuous Multi-Task Pre-training for Malicious URL Detection and Webpage Classification

arXiv: https://arxiv.org/abs/2402.11495 · HTML: https://mlsanity.com/paper/2402.11495/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 48 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [CharBERT: Character-aware Pre-trained Language Model](https://mlsanity.com/paper/2011.01513/) (2020)
- [URLNet: Learning a URL Representation with Deep Learning for Malicious URL Detection](https://mlsanity.com/paper/1802.03162/) (2018)
- [Malicious URL Detection using Machine Learning: A Survey](https://mlsanity.com/paper/1701.07179/) (2017)
- [PMANet: Malicious URL detection via post-trained language model guided multi-level feature attention network](https://mlsanity.com/paper/2311.12372/) (2023)
- [Look Before You Leap: Detecting Phishing Web Pages by Exploiting Raw URL And HTML Characteristics](https://mlsanity.com/paper/2011.04412/) (2020)
- [Toward More Generalized Malicious URL Detection Models](https://mlsanity.com/paper/2202.10027/) (2022)
- [TransURL: Improving malicious URL detection with multi-layer Transformer encoding and multi-scale pyramid features](https://mlsanity.com/paper/2312.00508/) (2023)
- [To Tune or Not To Tune? How About the Best of Both Worlds?](https://mlsanity.com/paper/1907.05338/) (2019)

## Built on by (most cited first, 3 of 3)

- [From Past to Present: A Survey of Malicious URL Detection Techniques, Datasets and Code Repositories](https://mlsanity.com/paper/2504.16449/) (2025)
- [MalURLBench: A Benchmark Evaluating Agents' Vulnerabilities When Processing Web URLs](https://mlsanity.com/paper/2601.18113/) (2026)
- [Beyond the Link: Assessing LLMs' ability to Classify Political Content across Global Media](https://mlsanity.com/paper/2506.17435/) (2025)

## Abstract (arXiv)

Malicious URL detection and webpage classification are critical tasks in cybersecurity and information management. In recent years, extensive research has explored using BERT or similar language models to replace traditional machine learning methods for detecting malicious URLs and classifying webpages. While previous studies show promising results, they often apply existing language models to these tasks without accounting for the inherent differences in domain data (e.g., URLs being loosely structured and semantically sparse compared to text), leaving room for performance improvement. Furthermore, current approaches focus on single tasks and have not been tested in multi-task scenarios. To address these challenges, we propose urlBERT, a pre-trained URL encoder leveraging Transformer to encode foundational knowledge from billions of unlabeled URLs. To achieve it, we propose to use 5 unsupervised pretraining tasks to capture multi-level information of URL lexical, syntax, and semantics, and generate contrastive and adversarial representations. Furthermore, to avoid inter-pre-training competition and interference, we proposed a grouped sequential learning method to ensure effective training across multi-tasks. Finally, we leverage a two-stage fine-tuning approach to improve the training stability and efficiency of the task model. To assess the multitasking potential of urlBERT, we fine-tune the task model in both single-task and multi-task modes. The former creates a classification model for a single task, while the latter builds a classification model capable of handling multiple tasks. We evaluate urlBERT on three downstream tasks: phishing URL detection, advertising URL detection, and webpage classification. The results demonstrate that urlBERT outperforms standard pre-trained models, and its multi-task mode is capable of addressing the real-world demands of multitasking.