# A Crawler Architecture for Harvesting the Clear, Social, and Dark Web for IoT-Related Cyber-Threat Intelligence

arXiv: https://arxiv.org/abs/2109.06932 · HTML: https://mlsanity.com/paper/2109.06932/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 39 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 1 of 1)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)

## Built on by (most cited first, 2 of 2)

- [ThreatCrawl: A BERT-based Focused Crawler for the Cybersecurity Domain](https://mlsanity.com/paper/2304.11960/) (2023)
- [CTI Dataset Construction from Telegram](https://mlsanity.com/paper/2509.20943/) (2025)

## Abstract (arXiv)

The clear, social, and dark web have lately been identified as rich sources of valuable cyber-security information that -given the appropriate tools and methods-may be identified, crawled and subsequently leveraged to actionable cyber-threat intelligence. In this work, we focus on the information gathering task, and present a novel crawling architecture for transparently harvesting data from security websites in the clear web, security forums in the social web, and hacker forums/marketplaces in the dark web. The proposed architecture adopts a two-phase approach to data harvesting. Initially a machine learning-based crawler is used to direct the harvesting towards websites of interest, while in the second phase state-of-the-art statistical language modelling techniques are used to represent the harvested information in a latent low-dimensional feature space and rank it based on its potential relevance to the task at hand. The proposed architecture is realised using exclusively open-source tools, and a preliminary evaluation with crowdsourced results demonstrates its effectiveness.