# THREAT/crawl: a Trainable, Highly-Reusable, and Extensible Automated Method and Tool to Crawl Criminal Underground Forums

arXiv: https://arxiv.org/abs/2212.03641 · HTML: https://mlsanity.com/paper/2212.03641/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2022.

## Built on by (most cited first, 2 of 2)

- [Cream Skimming the Underground: Identifying Relevant Information Points from Online Forums](https://mlsanity.com/paper/2308.02581/) (2023)
- [Inferring Discussion Topics about Exploitation of Vulnerabilities from Underground Hacking Forums](https://mlsanity.com/paper/2405.04561/) (2024)

## Abstract (arXiv)

Collecting data on underground criminal communities is highly valuable both for security research and security operations. Unfortunately these communities live within a constellation of diverse online forums that are difficult to infiltrate, may adopt crawling monitoring countermeasures, and require the development of ad-hoc scrapers for each different community, making the endeavour increasingly technically challenging, and potentially expensive. To address this problem we propose THREAT/crawl, a method and prototype tool for a highly reusable crawler that can learn a wide range of (arbitrary) forum structures, can remain under-the-radar during the crawling activity and can be extended and configured at the user will. We showcase THREAT/crawl capabilities and provide prime evaluation of our prototype against a range of active, live, underground communities.