# Web Content Filtering through knowledge distillation of Large Language Models

arXiv: https://arxiv.org/abs/2305.05027 · HTML: https://mlsanity.com/paper/2305.05027/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 8 May 2023.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://mlsanity.com/paper/1908.08962/) (2019)
- [Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics](https://mlsanity.com/paper/2110.01518/) (2021)
- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [URLNet: Learning a URL Representation with Deep Learning for Malicious URL Detection](https://mlsanity.com/paper/1802.03162/) (2018)
- [eXpose: A Character-Level Convolutional Neural Network with Embeddings For Detecting Malicious URLs, File Paths and Registry Keys](https://mlsanity.com/paper/1702.08568/) (2017)
- [URLTran: Improving Phishing URL Detection Using Transformers](https://mlsanity.com/paper/2106.05256/) (2021)
- [When Neural Networks Fail to Generalize? A Model Sensitivity Perspective](https://mlsanity.com/paper/2212.00850/) (2022)
- [Phishing URL Detection: A Network-based Approach Robust to Evasion](https://mlsanity.com/paper/2209.01454/) (2022)
- [Transformers for End-to-End InfoSec Tasks: A Feasibility Study](https://mlsanity.com/paper/2212.02666/) (2022)
- [Using Lexical Features for Malicious URL Detection -- A Machine Learning Approach](https://mlsanity.com/paper/1910.06277/) (2019)
- [Towards Web Phishing Detection Limitations and Mitigation](https://mlsanity.com/paper/2204.00985/) (2022)
- [Less is More: Robust and Novel Features for Malicious Domain Detection](https://mlsanity.com/paper/2006.01449/) (2020)
- [Training Transformers for Information Security Tasks: A Case Study on Malicious URL Prediction](https://mlsanity.com/paper/2011.03040/) (2020)
- [Semi-supervised Conditional GAN for Simultaneous Generation and Detection of Phishing URLs: A Game theoretic Perspective](https://mlsanity.com/paper/2108.01852/) (2021)
- [Exploring the Limits of Transfer Learning with Unified Model in the Cybersecurity Domain](https://mlsanity.com/paper/2302.10346/) (2023)
- [Detection of Malicious Websites Using Machine Learning Techniques](https://mlsanity.com/paper/2209.09630/) (2022)
- [Improving Web Content Blocking With Event-Loop-Turn Granularity JavaScript Signatures](https://mlsanity.com/paper/2005.11910/) (2020)
- [A Transferable and Automatic Tuning of Deep Reinforcement Learning for Cost Effective Phishing Detection](https://mlsanity.com/paper/2209.09033/) (2022)

## Built on by (most cited first, 7 of 7)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Large Language Models in Cybersecurity: State-of-the-Art](https://mlsanity.com/paper/2402.00891/) (2024)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generation](https://mlsanity.com/paper/2407.15748/) (2024)
- [Mapping the Landscape of Generative AI in Network Monitoring and Management](https://mlsanity.com/paper/2502.08576/) (2025)
- [AutoPureData: Automated Filtering of Undesirable Web Data to Update LLM Knowledge](https://mlsanity.com/paper/2406.19271/) (2024)
- [Research on a hybrid LSTM-CNN-Attention model for text-based web content classification](https://mlsanity.com/paper/2512.18475/) (2025)

## Abstract (arXiv)

We introduce a state-of-the-art approach for URL categorization that leverages the power of Large Language Models (LLMs) to address the primary objectives of web content filtering: safeguarding organizations from legal and ethical risks, limiting access to high-risk or suspicious websites, and fostering a secure and professional work environment. Our method utilizes LLMs to generate accurate classifications and then employs established knowledge distillation techniques to create smaller, more specialized student models tailored for web content filtering. Distillation results in a student model with a 9% accuracy rate improvement in classifying websites, sourced from customer telemetry data collected by a large security vendor, into 30 distinct content categories based on their URLs, surpassing the current state-of-the-art approach. Our student model matches the performance of the teacher LLM with 175 times less parameters, allowing the model to be used for in-line scanning of large volumes of URLs, and requires 3 orders of magnitude less manually labeled training data than the current state-of-the-art approach. Depending on the specific use case, the output generated by our approach can either be directly returned or employed as a pre-filter for more resource-intensive operations involving website images or HTML.