mlsanityData to 14 Oct 2026

ClueWeb22: 10 Billion Web Documents with Visual and Semantic Information

arXiv 2211.15848 ↗First posted 29 Nov 2022cs.IR cs.AI cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

ClueWeb22, the newest iteration of the ClueWeb line of datasets, provides 10 billion web pages affiliated with rich information. Its design was influenced by the need for a high quality, large scale web corpus to support a range of academic and industry research, for example, in information systems, retrieval-augmented AI systems, and model pretraining. Compared with earlier ClueWeb corpora, the ClueWeb22 corpus is larger, more varied, of higher-quality, and aligned with the document distributions in commercial web search. Besides raw HTML, ClueWeb22 includes rich information about the web pages provided by industry-standard document understanding systems, including the visual representation of pages rendered by a web browser, parsed HTML structure information from a neural network parser, and pre-processed cleaned document text to lower the barrier to entry. Many of these signals have been widely used in industry but are available to the research community for the first time at this scale.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

22 of 36 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
10 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  5. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  6. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  7. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  8. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  9. 2019Open Domain Web Keyphrase Extraction Beyond Language Modelingcited by 28
  10. 2020Selective Weak Supervision for Neural Information Retrievalcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 6; 2026: 4.

20 papers.

Sort:

By area:

  1. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  2. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 91
  3. 2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
  4. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 35
  5. 2023Sabi'a: Portuguese Large Language Modelscited by 19
  6. 2024Building a Large Japanese Web Corpus for Large Language Modelscited by 11
  7. 2024Synthetic Data Generation with Large Language Models for Personalized Community Question Answeringcited by 8
  8. 2024A Survey of Large Language Models for European Languagescited by 6
  9. 2026Fara-1.5: Scalable Learning Environments for Computer Use Agentscited by 6
  10. 2024Cleaner Pretraining Corpus Curation with Neural Web Scrapingcited by 5
  11. 2025Less LLM, More Documents: Searching for Improved RAGcited by 4
  12. 2025so much depends / upon / a whitespace: Why Whitespace Matters for Poets and LLMscited by 4
8 more
  1. 2026AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Webcited by 4
  2. 2025Revela: Dense Retriever Learning via Language Modelingcited by 3
  3. 2025Agent-centric Information Accesscited by 2
  4. 2023Unsupervised Dense Retrieval Training with Web Anchorscited by 1
  5. 2023Sequencing Matters: A Generate-Retrieve-Generate Model for Building Conversational Agentscited by 1
  6. 2025ORBIT -- Open Recommendation Benchmark for Reproducible Research with Hidden Testscited by 1
  7. 2026Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systemscited by 0
  8. 2026Host-Aware Web Crawling for LLM Pretraining Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026WCXB: A Multi-Type Web Content Extraction Benchmark
  2. 2025Going over Fine Web with a Fine-Tooth Comb: Technical Report of Indexing Fine Web for Problematic Content Search and Retrieval
  3. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
  4. 2026LiveWeb-IE: A Benchmark For Online Web Information Extraction
  5. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search
  6. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs
  7. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models
  8. 2023Text Classification via Large Language Models