# Newswire: A Large-Scale Structured Database of a Century of Historical News

arXiv: https://arxiv.org/abs/2406.09490 · HTML: https://mlsanity.com/paper/2406.09490/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 18 of 18)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [ReFinED: An Efficient Zero-shot-capable Approach to End-to-End Entity Linking](https://mlsanity.com/paper/2207.04108/) (2022)
- [Do Language Models Plagiarize?](https://mlsanity.com/paper/2203.07618/) (2022)
- [Noise-Robust De-Duplication at Scale](https://mlsanity.com/paper/2210.04261/) (2022)
- [American Stories: A Large-Scale Structured Text Dataset of Historical U.S. Newspapers](https://mlsanity.com/paper/2308.12477/) (2023)
- [Global Entity Disambiguation with BERT](https://mlsanity.com/paper/1909.00426/) (2019)
- [Efficient OCR for Building a Diverse Digital History](https://mlsanity.com/paper/2304.02737/) (2023)
- [EfficientOCR: An Extensible, Open-Source Package for Efficiently Digitizing World Knowledge](https://mlsanity.com/paper/2310.10050/) (2023)
- [A Massive Scale Semantic Similarity Dataset of Historical English](https://mlsanity.com/paper/2306.17810/) (2023)
- [OLALA: Object-Level Active Learning for Efficient Document Layout Annotation](https://mlsanity.com/paper/2010.01762/) (2020)

## Built on by (most cited first, 5 of 5)

- [It's High Time: A Survey of Temporal Question Answering](https://mlsanity.com/paper/2505.20243/) (2025)
- [Contrastive Entity Coreference and Disambiguation for Historical Texts](https://mlsanity.com/paper/2406.15576/) (2024)
- [NewsHomepages: Homepage Layouts Capture Information Prioritization Decisions](https://mlsanity.com/paper/2501.00004/) (2024)
- [Southern Newswires: A Large-Scale Study of Mid-Century Wire Content Beyond the Front Page](https://mlsanity.com/paper/2502.11866/) (2025)
- [Chronos: The AI Co-Historian](https://mlsanity.com/paper/2604.03553/) (2026)

## Abstract (arXiv)

In the U.S. historically, local newspapers drew their content largely from newswires like the Associated Press. Historians argue that newswires played a pivotal role in creating a national identity and shared understanding of the world, but there is no comprehensive archive of the content sent over newswires. We reconstruct such an archive by applying a customized deep learning pipeline to hundreds of terabytes of raw image scans from thousands of local newspapers. The resulting dataset contains 2.7 million unique public domain U.S. newswire articles, written between 1878 and 1977. Locations in these articles are georeferenced, topics are tagged using customized neural topic classification, named entities are recognized, and individuals are disambiguated to Wikipedia using a novel entity disambiguation model. To construct the Newswire dataset, we first recognize newspaper layouts and transcribe around 138 millions structured article texts from raw image scans. We then use a customized neural bi-encoder model to de-duplicate reproduced articles, in the presence of considerable abridgement and noise, quantifying how widely each article was reproduced. A text classifier is used to ensure that we only include newswire articles, which historically are in the public domain. The structured data that accompany the texts provide rich information about the who (disambiguated individuals), what (topics), and where (georeferencing) of the news that millions of Americans read over the course of a century. We also include Library of Congress metadata information about the newspapers that ran the articles on their front pages. The Newswire dataset is useful both for large language modeling - expanding training data beyond what is available from modern web texts - and for studying a diversity of questions in computational linguistics, social science, and the digital humanities.