# Web2Text: Deep Structured Boilerplate Removal

arXiv: https://arxiv.org/abs/1801.02607 · HTML: https://mlsanity.com/paper/1801.02607/

## Key facts

- Cited by 10 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 8 Jan 2018.

## Built on by (most cited first, 10 of 10)

- [Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM](https://mlsanity.com/paper/2511.23119/) (2025)
- [Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions](https://mlsanity.com/paper/2403.15279/) (2024)
- [ProVe: A Pipeline for Automated Provenance Verification of Knowledge Graphs against Textual Sources](https://mlsanity.com/paper/2210.14846/) (2022)
- [Cleaner Pretraining Corpus Curation with Neural Web Scraping](https://mlsanity.com/paper/2402.14652/) (2024)
- [Enhancing Mobile "How-to" Queries with Automated Search Results Verification and Reranking](https://mlsanity.com/paper/2404.08860/) (2024)
- [WebLists: Extracting Structured Information From Complex Interactive Websites Using Executable LLM Agents](https://mlsanity.com/paper/2504.12682/) (2025)
- [Boilerplate Detection via Semantic Classification of TextBlocks](https://mlsanity.com/paper/2203.04467/) (2022)
- [GROWN+UP: A Graph Representation Of a Webpage Network Utilizing Pre-training](https://mlsanity.com/paper/2208.02252/) (2022)
- [AXE: Low-Cost Cross-Domain Web Structured Information Extraction](https://mlsanity.com/paper/2602.01838/) (2026)
- [WCXB: A Multi-Type Web Content Extraction Benchmark](https://mlsanity.com/paper/2605.21097/) (2026)

## Abstract (arXiv)

Web pages are a valuable source of information for many natural language processing and information retrieval tasks. Extracting the main content from those documents is essential for the performance of derived applications. To address this issue, we introduce a novel model that performs sequence labeling to collectively classify all text blocks in an HTML page as either boilerplate or main content. Our method uses a hidden Markov model on top of potentials derived from DOM tree features using convolutional neural networks. The proposed method sets a new state-of-the-art performance for boilerplate removal on the CleanEval benchmark. As a component of information retrieval pipelines, it improves retrieval performance on the ClueWeb12 collection.