# DarkBERT: A Language Model for the Dark Side of the Internet

arXiv: https://arxiv.org/abs/2305.08596 · HTML: https://mlsanity.com/paper/2305.08596/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 30 references that are in the map.
- First posted on arXiv 15 May 2023.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?](https://mlsanity.com/paper/2104.07762/) (2021)
- [Self-Supervised Euphemism Detection and Identification for Content Moderation](https://mlsanity.com/paper/2103.16808/) (2021)
- [The Dark Side of the Language: Pre-trained Transformers in the DarkNet](https://mlsanity.com/paper/2201.05613/) (2022)
- [The Language of Legal and Illegal Activity on the Darknet](https://mlsanity.com/paper/1905.05543/) (2019)
- [Shedding New Light on the Language of the Dark Web](https://mlsanity.com/paper/2204.06885/) (2022)
- [Spoiled Onions: Exposing Malicious Tor Exit Relays](https://mlsanity.com/paper/1401.4917/) (2014)
- [Avaddon ransomware: an in-depth analysis and decryption of infected systems](https://mlsanity.com/paper/2102.04796/) (2021)

## Built on by (most cited first, 13 of 13)

- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Anatomy of an AI-powered malicious social botnet](https://mlsanity.com/paper/2307.16336/) (2023)
- [LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks](https://mlsanity.com/paper/2310.11409/) (2023)
- [KidLM: Advancing Language Models for Children -- Early Insights and Future Directions](https://mlsanity.com/paper/2410.03884/) (2024)
- [RedTeamLLM: an Agentic AI framework for offensive security](https://mlsanity.com/paper/2505.06913/) (2025)
- [Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain](https://mlsanity.com/paper/2403.10576/) (2024)
- [Towards a Holodeck-style Simulation Game](https://mlsanity.com/paper/2308.13548/) (2023)
- [EventHunter: Dynamic Clustering and Ranking of Security Events from Hacker Forum Discussions](https://mlsanity.com/paper/2507.09762/) (2025)
- [ClausewitzGPT Framework: A New Frontier in Theoretical Large Language Model Enhanced Information Operations](https://mlsanity.com/paper/2310.07099/) (2023)
- [Two-step Automated Cybercrime Coded Word Detection using Multi-level Representation Learning](https://mlsanity.com/paper/2403.10838/) (2024)
- [Responsible AI in NLP: GUS-Net Span-Level Bias Detection Dataset and Benchmark for Generalizations, Unfairness, and Stereotypes](https://mlsanity.com/paper/2410.08388/) (2024)
- [Covering Cracks in Content Moderation: Delexicalized Distant Supervision for Illicit Drug Jargon Detection](https://mlsanity.com/paper/2503.14926/) (2025)
- [A Language Model-Driven Semi-Supervised Ensemble Framework for Illicit Market Detection Across Deep/Dark Web and Social Platforms](https://mlsanity.com/paper/2507.22912/) (2025)

## Abstract (arXiv)

Recent research has suggested that there are clear differences in the language used in the Dark Web compared to that of the Surface Web. As studies on the Dark Web commonly require textual analysis of the domain, language models specific to the Dark Web may provide valuable insights to researchers. In this work, we introduce DarkBERT, a language model pretrained on Dark Web data. We describe the steps taken to filter and compile the text data used to train DarkBERT to combat the extreme lexical and structural diversity of the Dark Web that may be detrimental to building a proper representation of the domain. We evaluate DarkBERT and its vanilla counterpart along with other widely used language models to validate the benefits that a Dark Web domain specific model offers in various use cases. Our evaluations show that DarkBERT outperforms current language models and may serve as a valuable resource for future research on the Dark Web.