mlsanityData to 7 Oct 2026

Simple and Efficient ways to Improve REALM

arXiv 2104.08710 ↗First posted 18 Apr 2021cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Dense retrieval has been shown to be effective for retrieving relevant documents for Open Domain QA, surpassing popular sparse retrieval methods like BM25. REALM (Guu et al., 2020) is an end-to-end dense retrieval system that relies on MLM based pretraining for improved downstream QA efficiency across multiple datasets. We study the finetuning of REALM on various QA tasks and explore the limits of various hyperparameter and supervision choices. We find that REALM was significantly undertrained when finetuning and simple improvements in the training, supervision, and inference setups can significantly benefit QA results and exceed the performance of other models published post it. Our best model, REALM++, incorporates all the best working findings and achieves significant QA accuracy improvements over baselines (~5.5% absolute accuracy) without any model design changes. Additionally, REALM++ matches the performance of large Open Domain QA models which have 3x more parameters demonstrating the efficiency of the setup.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

20 of 27 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  7. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  8. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  10. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  11. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  12. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
8 more
  1. 2019A Discrete Hard EM Approach for Weakly Supervised Question Answeringcited by 78
  2. 2019Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Indexcited by 64
  3. 2019Knowledge Guided Text Retrieval and Reading for Open Domain Question Answeringcited by 59
  4. 2019Benchmarking TPU, GPU, and CPU Platforms for Deep Learningcited by 51
  5. 2019Revealing the Importance of Semantic Retrieval for Machine Reading at Scalecited by 50
  6. 2019Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answeringcited by 48
  7. 2018Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehensioncited by 23
  8. 2020RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answeringcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 1.

3 papers.

  1. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  2. 2021You Only Need One Model for Open-domain Question Answeringcited by 17
  3. 2022FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answeringcited by 5