# ClueWeb22: 10 Billion Web Documents with Visual and Semantic Information

arXiv: https://arxiv.org/abs/2211.15848 · HTML: https://mlsanity.com/paper/2211.15848/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus](https://mlsanity.com/paper/2112.09924/) (2021)
- [Open Domain Web Keyphrase Extraction Beyond Language Modeling](https://mlsanity.com/paper/1911.02671/) (2019)
- [Selective Weak Supervision for Neural Information Retrieval](https://mlsanity.com/paper/2001.10382/) (2020)

## Built on by (most cited first, 20 of 20)

- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [CodeRAG-Bench: Can Retrieval Augment Code Generation?](https://mlsanity.com/paper/2406.14497/) (2024)
- [Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale](https://mlsanity.com/paper/2409.15637/) (2024)
- [Fara-7B: An Efficient Agentic Model for Computer Use](https://mlsanity.com/paper/2511.19663/) (2025)
- [Sabi'a: Portuguese Large Language Models](https://mlsanity.com/paper/2304.07880/) (2023)
- [Building a Large Japanese Web Corpus for Large Language Models](https://mlsanity.com/paper/2404.17733/) (2024)
- [Synthetic Data Generation with Large Language Models for Personalized Community Question Answering](https://mlsanity.com/paper/2410.22182/) (2024)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Fara-1.5: Scalable Learning Environments for Computer Use Agents](https://mlsanity.com/paper/2606.20785/) (2026)
- [Cleaner Pretraining Corpus Curation with Neural Web Scraping](https://mlsanity.com/paper/2402.14652/) (2024)
- [Less LLM, More Documents: Searching for Improved RAG](https://mlsanity.com/paper/2510.02657/) (2025)
- [so much depends / upon / a whitespace: Why Whitespace Matters for Poets and LLMs](https://mlsanity.com/paper/2510.16713/) (2025)
- [AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web](https://mlsanity.com/paper/2604.10938/) (2026)
- [Revela: Dense Retriever Learning via Language Modeling](https://mlsanity.com/paper/2506.16552/) (2025)
- [Agent-centric Information Access](https://mlsanity.com/paper/2502.19298/) (2025)
- [Unsupervised Dense Retrieval Training with Web Anchors](https://mlsanity.com/paper/2305.05834/) (2023)
- [Sequencing Matters: A Generate-Retrieve-Generate Model for Building Conversational Agents](https://mlsanity.com/paper/2311.09513/) (2023)
- [ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Tests](https://mlsanity.com/paper/2510.26095/) (2025)
- [Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems](https://mlsanity.com/paper/2609.08887/) (2026)
- [Host-Aware Web Crawling for LLM Pretraining Data](https://mlsanity.com/paper/2610.13321/) (2026)

## Abstract (arXiv)

ClueWeb22, the newest iteration of the ClueWeb line of datasets, provides 10 billion web pages affiliated with rich information. Its design was influenced by the need for a high quality, large scale web corpus to support a range of academic and industry research, for example, in information systems, retrieval-augmented AI systems, and model pretraining. Compared with earlier ClueWeb corpora, the ClueWeb22 corpus is larger, more varied, of higher-quality, and aligned with the document distributions in commercial web search. Besides raw HTML, ClueWeb22 includes rich information about the web pages provided by industry-standard document understanding systems, including the visual representation of pages rendered by a web browser, parsed HTML structure information from a neural network parser, and pre-processed cleaned document text to lower the barrier to entry. Many of these signals have been widely used in industry but are available to the research community for the first time at this scale.