mlsanityData to 14 Oct 2026

The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus

arXiv 2112.09924 ↗First posted 18 Dec 2021cs.CL cs.AI cs.IRcited by 60

Not on the map. Why?

Abstract (arXiv)

In order to address increasing demands of real-world applications, the research for knowledge-intensive NLP (KI-NLP) should advance by capturing the challenges of a truly open-domain environment: web-scale knowledge, lack of structure, inconsistent quality and noise. To this end, we propose a new setup for evaluating existing knowledge intensive tasks in which we generalize the background corpus to a universal web snapshot. We investigate a slate of NLP tasks which rely on knowledge - either factual or common sense, and ask systems to use a subset of CCNet - the Sphere corpus - as a knowledge source. In contrast to Wikipedia, otherwise a common background corpus in KI-NLP, Sphere is orders of magnitude larger and better reflects the full diversity of knowledge on the web. Despite potential gaps in coverage, challenges of scale, lack of structure and lower quality, we find that retrieval from Sphere enables a state of the art system to match and even outperform Wikipedia-based models on several tasks. We also observe that while a dense index can outperform a sparse BM25 baseline on Wikipedia, on Sphere this is not yet possible. To facilitate further research and minimise the community's reliance on proprietary, black-box search engines, we share our indices, evaluation metrics and infrastructure.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Builds on 53 of its 69 references that are in the map. First posted on arXiv 18 Dec 2021.

Builds on

53 of 69 references.

References · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2017Billion-scale similarity search with GPUscited by 2,497
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  11. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
38 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  3. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  4. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  5. 2019ELI5: Long Form Question Answeringcited by 575
  6. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  7. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  8. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  9. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  10. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  11. 2019Abductive Commonsense Reasoningcited by 301
  12. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 285
  13. 2021Internet-Augmented Dialogue Generationcited by 217
  14. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 214
  15. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 200
  16. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  17. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  18. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  19. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  20. 2020Pre-training via Paraphrasingcited by 92
  21. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 80
  22. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  23. 2021Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representationscited by 60
  24. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 60
  25. 2021Efficient Passage Retrieval with Hashing for Open-domain Question Answeringcited by 53
  26. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  27. 2020GenericsKB: A Knowledge Base of Generic Statementscited by 51
  28. 2021Multi-task Retrieval for Knowledge-Intensive Taskscited by 44
  29. 2021Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attentioncited by 40
  30. 2021Hindsight: Posterior-guided training of retrievers for improved open-ended generationcited by 38
  31. 2021Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performancecited by 35
  32. 2015Quantization based Fast Inner Product Searchcited by 29
  33. 2020Differentiable Open-Ended Commonsense Reasoningcited by 29
  34. 2021Robust Retrieval Augmented Generation for Zero-shot Slot Fillingcited by 24
  35. 2019How Additional Knowledge can Improve Natural Language Commonsense Question Answering?cited by 22
  36. 2016Women Through the Glass Ceiling: Gender Asymmetries in Wikipediacited by 17
  37. 2021CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Trainingcited by 13
  38. 2015First Women, Second Sex: Gender Bias in Wikipediacited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 17; 2023: 16; 2024: 19; 2025: 8.

60 papers.

Sort:

By area:

  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  2. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  3. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  4. 2023Tool Learning with Foundation Modelscited by 302
  5. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  6. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  7. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  8. 2024Corrective Retrieval Augmented Generationcited by 197
  9. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  10. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
  11. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 113
  12. 2022Task-aware Retrieval with Instructionscited by 107
38 more
  1. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  2. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 97
  3. 2022PEER: A Collaborative Language Modelcited by 86
  4. 2024Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbationscited by 65
  5. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  6. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  7. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  8. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  9. 2024Training Language Models to Generate Text with Citations via Fine-grained Rewardscited by 42
  10. 2024Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refusecited by 34
  11. 2023WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpuscited by 29
  12. 2024Learning to Plan and Generate Text with Citationscited by 28
  13. 2024Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generationcited by 25
  14. 2024Jill Watson: A Virtual Teaching Assistant powered by ChatGPTcited by 24
  15. 2024Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generationcited by 21
  16. 2022ClueWeb22: 10 Billion Web Documents with Visual and Semantic Informationcited by 20
  17. 2023Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!cited by 19
  18. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inferencecited by 15
  19. 2024Advancing Large Language Model Attribution through Self-Improvingcited by 14
  20. 2024Learning Fine-Grained Grounded Citations for Attributed Large Language Modelscited by 13
  21. 2025Learning Facts at Scale with Active Readingcited by 13
  22. 2022Cross-document Event Coreference Search: Task, Dataset and Modelingcited by 12
  23. 2022Improving Wikipedia Verifiability with AIcited by 11
  24. 2023GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Rankingcited by 11
  25. 2024CaLM: Contrasting Large and Small Language Models to Verify Grounded Generationcited by 11
  26. 2024Open-World Evaluation for Retrieving Diverse Perspectivescited by 11
  27. 2022Multi-Task Retrieval-Augmented Text Generation with Relevance Samplingcited by 10
  28. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  29. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  30. 2025Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarkscited by 8
  31. 2022Generating Full Length Wikipedia Biographies: The Impact of Gender Bias on the Retrieval-Based Generation of Women Biographiescited by 7
  32. 2023The Web Can Be Your Oyster for Improving Large Language Modelscited by 7
  33. 2023Specious Sites: Tracking the Spread and Sway of Spurious News Stories at Scalecited by 7
  34. 2024Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generationcited by 6
  35. 2024Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modelingcited by 6
  36. 2023Ladder-of-Thought: Using Knowledge as Steps to Elevate Stance Detectioncited by 4
  37. 2022KGI: An Integrated Framework for Knowledge Intensive Language Taskscited by 3
  38. 2025MegaWika 2: A More Comprehensive Multilingual Collection of Articles and their Sourcescited by 3

50 most cited of 60 All 60 (JSON)