# Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions

arXiv: https://arxiv.org/abs/2403.15279 · HTML: https://mlsanity.com/paper/2403.15279/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 24 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 6 of 6)

- [Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection](https://mlsanity.com/paper/2201.10474/) (2022)
- [OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMs](https://mlsanity.com/paper/2309.03876/) (2023)
- [Hate is the New Infodemic: A Topic-aware Modeling of Hate Speech Diffusion on Twitter](https://mlsanity.com/paper/2010.04377/) (2020)
- [Boilerplate Removal using a Neural Sequence Labeling Model](https://mlsanity.com/paper/2004.14294/) (2020)
- [Web2Text: Deep Structured Boilerplate Removal](https://mlsanity.com/paper/1801.02607/) (2018)
- [FastWARC: Optimizing Large-Scale Web Archive Analytics](https://mlsanity.com/paper/2112.03103/) (2021)

## Built on by (most cited first, 7 of 7)

- [Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMs](https://mlsanity.com/paper/2410.10739/) (2024)
- [WebLists: Extracting Structured Information From Complex Interactive Websites Using Executable LLM Agents](https://mlsanity.com/paper/2504.12682/) (2025)
- [Infini-News: Efficiently Queryable Access to 1.3 Billion Processed Common Crawl News Articles](https://mlsanity.com/paper/2605.18337/) (2026)
- [Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling](https://mlsanity.com/paper/2604.28075/) (2026)
- [news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling](https://mlsanity.com/paper/2607.21284/) (2026)
- [Global Crises and National Policies: A Large Scale Analysis of Political Content in German Language Online Media](https://mlsanity.com/paper/2608.18268/) (2026)
- [PACE: Publisher-Adaptive Content Extraction via Agentic Automation](https://mlsanity.com/paper/2608.27466/) (2026)

## Abstract (arXiv)

This paper introduces Fundus, a user-friendly news scraper that enables users to obtain millions of high-quality news articles with just a few lines of code. Unlike existing news scrapers, we use manually crafted, bespoke content extractors that are specifically tailored to the formatting guidelines of each supported online newspaper. This allows us to optimize our scraping for quality such that retrieved news articles are textually complete and without HTML artifacts. Further, our framework combines both crawling (retrieving HTML from the web or large web archives) and content extraction into a single pipeline. By providing a unified interface for a predefined collection of newspapers, we aim to make Fundus broadly usable even for non-technical users. This paper gives an overview of the framework, discusses our design choices, and presents a comparative evaluation against other popular news scrapers. Our evaluation shows that Fundus yields significantly higher quality extractions (complete and artifact-free news articles) than prior work. The framework is available on GitHub under https://github.com/flairNLP/fundus and can be simply installed using pip.