mlsanityData to 7 Oct 2026

Artefact Retrieval: Overview of NLP Models with Knowledge Base Access

arXiv 2201.09651 ↗First posted 24 Jan 2022cs.CL cs.IRcited by 4

Not on the map. Why?

Abstract (arXiv)

Many NLP models gain performance by having access to a knowledge base. A lot of research has been devoted to devising and improving the way the knowledge base is accessed and incorporated into the model, resulting in a number of mechanisms and pipelines. Despite the diversity of proposed mechanisms, there are patterns in the designs of such systems. In this paper, we systematically describe the typology of artefacts (items retrieved from a knowledge base), retrieval mechanisms and the way these artefacts are fused into the model. This further allows us to uncover combinations of design decisions that had not yet been tried. Most of the focus is given to language models, though we also show how question answering, fact-checking and knowledgable dialogue models fit into this system as well. Having an abstract model which can describe the architecture of specific models also helps with transferring these architectures between multiple NLP tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 59 of its 91 references that are in the map. First posted on arXiv 24 Jan 2022.

Builds on

59 of 91 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
38 more
  1. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  2. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  3. 2019How Can We Know What Language Models Know?cited by 953
  4. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  5. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  6. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  7. 2014Distributed Representations of Sentences and Documentscited by 677
  8. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  9. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  10. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  11. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  12. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  13. 2014Memory Networkscited by 354
  14. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  15. 2020Autoregressive Entity Retrievalcited by 333
  16. 2018Analyzing Inverse Problems with Invertible Neural Networkscited by 239
  17. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  18. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  19. 2019Multi-Stage Document Ranking with BERTcited by 200
  20. 2021How Many Data Points is a Prompt Worth?cited by 197
  21. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  22. 2019PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Textcited by 182
  23. 2018Open Domain Question Answering Using Early Fusion of Knowledge Bases and Textcited by 167
  24. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  25. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  26. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  27. 2015Large-scale Simple Question Answering with Memory Networkscited by 127
  28. 2018Combining Fact Extraction and Verification with Neural Semantic Matching Networkscited by 120
  29. 2021Case-based Reasoning for Natural Language Queries over Knowledge Basescited by 107
  30. 2021Adaptive Semiparametric Language Modelscited by 83
  31. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  32. 2019Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modelingcited by 71
  33. 2020Understanding and Mitigating Exploding Inverses in Invertible Neural Networkscited by 71
  34. 2019BERT for Evidence Retrieval and Claim Verificationcited by 57
  35. 2020Language Models as Fact Checkers?cited by 51
  36. 2020The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizescited by 48
  37. 2021Multi-task Retrieval for Knowledge-Intensive Taskscited by 44
  38. 2020Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Primingcited by 41

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 0; 2024: 1.

4 papers.

  1. 2022A Survey on Medical Document Summarizationcited by 14
  2. 2022Knowledge Base Index Compression via Dimensionality and Precision Reductioncited by 2
  3. 2022Fusing Sentence Embeddings Into LSTM-based Autoregressive Language Modelscited by 1
  4. 2024Synergizing Machine Learning & Symbolic Methods: A Survey on Hybrid Approaches to Natural Language Processingcited by 1