# Infini-News: Efficiently Queryable Access to 1.3 Billion Processed Common Crawl News Articles

arXiv: https://arxiv.org/abs/2605.18337 · HTML: https://mlsanity.com/paper/2605.18337/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 62 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 14 of 14)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Framing and Agenda-setting in Russian News: a Computational Analysis of Intricate Political Strategies](https://mlsanity.com/paper/1808.09386/) (2018)
- [NELA-GT-2018: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles](https://mlsanity.com/paper/1904.01546/) (2019)
- [Media Cloud: Massive Open Source Collection of Global News on the Open Web](https://mlsanity.com/paper/2104.03702/) (2021)
- [FastWARC: Optimizing Large-Scale Web Archive Analytics](https://mlsanity.com/paper/2112.03103/) (2021)
- [Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions](https://mlsanity.com/paper/2403.15279/) (2024)
- [UKElectionNarratives: A Dataset of Misleading Narratives Surrounding Recent UK General Elections](https://mlsanity.com/paper/2505.05459/) (2025)
- [Mapping the Global Election Landscape on Social Media in 2024](https://mlsanity.com/paper/2406.04962/) (2024)
- [TeleScope: A Longitudinal Dataset for Investigating Online Discourse and Information Interaction on Telegram](https://mlsanity.com/paper/2504.19536/) (2025)
- [Big Data Science Over the Past Web](https://mlsanity.com/paper/2108.01605/) (2021)

## Built on by (most cited first, 1 of 1)

- [Mapping Political-Elite Networks in Europe with a Multilingual Joint Entity-Relation Extraction Pipeline](https://mlsanity.com/paper/2606.27347/) (2026)

## Abstract (arXiv)

Large-scale news corpora support a wide range of research in Computational Social Science and NLP, yet access remains constrained: commercial archives impose prohibitive costs and licensing restrictions, while open alternatives like Common Crawl's CC-News require terabyte-scale storage and computationally intensive processing. We present Infini-News, a retrieval toolkit and index for the entire CC-News archive from August 2016 to the latest available snapshot. Our contributions are threefold. First, we extract, clean the text, and parse the structured metadata of over 1.35B articles. Second, we enrich the corpus with language detection using three frontier language classifiers (GlotLID, lingua, and CommonLingua), and with multi-source geographic attribution that resolves a country of origin for 83.4% of articles across 222 countries. Third, we construct Infini-gram indexes: suffix-array structures that let researchers search the full archive for arbitrary text patterns in sub-second time. Together, these resources lower the barrier to longitudinal, cross-national media research.