mlsanityData to 14 Oct 2026

CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training

arXiv 2110.07731 ↗First posted 14 Oct 2021cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

With the rise of large-scale pre-trained language models, open-domain question-answering (ODQA) has become an important research topic in NLP. Based on the popular pre-training fine-tuning approach, we posit that an additional in-domain pre-training stage using a large-scale, natural, and diverse question-answering (QA) dataset can be beneficial for ODQA. Consequently, we propose a novel QA dataset based on the Common Crawl project in this paper. Using the readily available schema.org annotation, we extract around 130 million multilingual question-answer pairs, including about 60 million English data-points. With this previously unseen number of natural QA pairs, we pre-train popular language models to show the potential of large-scale in-domain pre-training for the task of question-answering. In our experiments, we find that pre-training question-answering models on our Common Crawl Question Answering dataset (CCQA) achieves promising results in zero-shot, low resource and fine-tuned settings across multiple tasks, models and benchmarks.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 27 of its 31 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

27 of 31 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  10. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  11. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
15 more
  1. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  2. 2019ELI5: Long Form Question Answeringcited by 575
  3. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  4. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  5. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  6. 2016FastText.zip: Compressing text classification modelscited by 335
  7. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 200
  8. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  9. 2020Answering Complex Open-Domain Questions with Multi-Hop Dense Retrievalcited by 162
  10. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  11. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 138
  12. 2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
  13. 2021GooAQ: Open Question Answering with Diverse Answer Typescited by 53
  14. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  15. 2021Question Answering Infused Pre-training of General-Purpose Contextualized Representationscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 3; 2024: 2; 2025: 1; 2026: 2.

13 papers.

Sort:

  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  2. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  3. 2021The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpuscited by 60
  4. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28
  5. 2023CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Datacited by 21
  6. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  7. 2025WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrievalcited by 10
  8. 2024Generative Explore-Exploit: Training-free Optimization of Generative Recommender Systems using LLM Optimizerscited by 7
  9. 2022Few-shot Adaptation Works with UnpredicTable Datacited by 6
  10. 2022Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysiscited by 5
  11. 2026WebFAQ 2.0: A Multilingual QA Dataset with Mined Hard Negatives for Dense Retrievalcited by 1
  12. 2024CIKMar: A Dual-Encoder Approach to Prompt-Based Reranking in Educational Dialogue Systemscited by 0
1 more
  1. 2026You Can Tell Who's Asking: What the Web's Questions Are Made Of, and Where They Come Fromcited by 0