mlsanityData to 7 Oct 2026

Variational Open-Domain Question Answering

arXiv 2210.06345 ↗First posted 23 Sep 2022cs.CL cs.IR cs.LGcited by 5

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-augmented models have proven to be effective in natural language processing tasks, yet there remains a lack of research on their optimization using variational inference. We introduce the Variational Open-Domain (VOD) framework for end-to-end training and evaluation of retrieval-augmented models, focusing on open-domain question answering and language modelling. The VOD objective, a self-normalized estimate of the R'enyi variational bound, approximates the task marginal likelihood and is evaluated under samples drawn from an auxiliary sampling distribution (cached retriever and/or approximate posterior). It remains tractable, even for retriever distributions defined on large corpora. We demonstrate VOD's versatility by training reader-retriever BERT-sized models on multiple-choice medical exam questions. On the MedMCQA dataset, we outperform the domain-tuned Med-PaLM by +5.3% despite using 2.500× fewer parameters. Our retrieval-augmented BioLinkBERT model scored 62.9% on the MedMCQA and 55.0% on the MedQA-USMLE. Last, we show the effectiveness of our learned retriever component in the context of medical semantic search.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 51 of its 73 references that are in the map. First posted on arXiv 23 Sep 2022.

Builds on

51 of 73 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
36 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2017Billion-scale similarity search with GPUscited by 2,477
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  12. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  13. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  14. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  15. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  16. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  17. 2015Importance Weighted Autoencoderscited by 454
  18. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  19. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  20. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  21. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 205
  22. 2022Can large language models reason about medical questions?cited by 194
  23. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  24. 2018Inference Suboptimality in Variational Autoencoderscited by 131
  25. 2014Neural Variational Inference and Learning in Belief Networkscited by 116
  26. 2016R'enyi Divergence Variational Inferencecited by 104
  27. 2017Backpropagation through the Void: Optimizing control variates for black-box gradient estimationcited by 104
  28. 2016Variational inference for Monte Carlo objectivescited by 83
  29. 2017REBAR: Low-variance, unbiased gradient estimates for discrete latent variable modelscited by 78
  30. 2018Tighter Variational Bounds are Not Necessarily Bettercited by 76
  31. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71
  32. 2021Hindsight: Posterior-guided training of retrievers for improved open-ended generationcited by 38
  33. 2020Is Retriever Merely an Approximator of Reader?cited by 30
  34. 2019The Thermodynamic Variational Objectivecited by 26
  35. 2020Optimal Variance Control of the Score Function Gradient Estimator for Importance Weighted Boundscited by 7
  36. 2013FindZebra: A search engine for rare diseasescited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3.

5 papers.

  1. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 62
  2. 2023Enhancing Small Medical Learners with Privacy-preserving Contextual Promptingcited by 11
  3. 2024Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trendscited by 3
  4. 2024MedMobile: A mobile-sized language model with clinical capabilitiescited by 3
  5. 2024A Statistical Framework for Data-dependent Retrieval-Augmented Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale
  2. 2024Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts
  3. 2023BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer
  4. 2024Answerability in Retrieval-Augmented Open-Domain Question Answering
  5. 2024Towards Better Generalization in Open-Domain Question Answering by Mitigating Context Memorization
  6. 2021End-to-End Training of Neural Retrievers for Open-Domain Question Answering
  7. 2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering
  8. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Context