No News is Good News: A Critique of the One Billion Word Benchmark
Not on the map. Why?
Abstract (arXiv)
The One Billion Word Benchmark is a dataset derived from the WMT 2011 News Crawl, commonly used to measure language modeling ability in natural language processing. We train models solely on Common Crawl web scrapes partitioned by year, and demonstrate that they perform worse on this task over time due to distributional shift. Analysis of this corpus reveals that it contains several examples of harmful text, as well as outdated references to current events. We suggest that the temporal nature of news and its distribution shift over time makes it poorly suited for measuring language modeling ability, and discuss potential impact and considerations for researchers building language models and evaluation datasets.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 10 of its 14 references that are in the map. First posted on arXiv 25 Oct 2021.
Builds on
10 of 14 references.
References · 10
- 2017Attention Is All You Needcited by 60,106
- 2020Language Models are Few-Shot Learnerscited by 32,320
- 2020Scaling Laws for Neural Language Modelscited by 6,494
- 2019Defending Against Neural Fake Newscited by 559
- 2017Fake News Detection on Social Media: A Data Mining Perspectivecited by 335
- 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 306
- 2021Changing the World by Changing the Datacited by 51
- 2021Mitigating harm in language models with conditional-likelihood filtrationcited by 39
- 2021A Survey on Predicting the Factuality and the Bias of News Mediacited by 20
- 2013Characterizing the Life Cycle of Online News Stories Using Social Media Reactionscited by 7
Built on by
Not cited yet.