Exploratory Analysis of a Terabyte Scale Web Corpus
Not on the map. Why?
Abstract (arXiv)
In this paper we present a preliminary analysis over the largest publicly accessible web dataset: the Common Crawl Corpus. We measure nine web characteristics from two levels of granularity using MapReduce and we comment on the initial observations over a fraction of it. To the best of our knowledge two of the characteristics, the language distribution and the HTML version of pages have not been analyzed in previous work, while the specific dataset has been only analyzed on page level.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Sep 2014.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 1; 2022: 1; 2023: 0; 2024: 1.
3 papers.
- 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 80
- 2022Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performancecited by 11
- 2024Beyond the Safety Bundle: Auditing the Helpful and Harmless Datasetcited by 8
- 2024Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset12 Nov 2024
- 2022Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance4 May 2022
- 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus6 May 2021