mlsanityData to 7 Oct 2026

NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

arXiv 2101.00133 ↗First posted 1 Jan 2021cs.CL cs.AIcited by 52

Not on the map. Why?

Abstract (arXiv)

We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 45 of its 68 references that are in the map. First posted on arXiv 1 Jan 2021.

Builds on

45 of 68 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2017Billion-scale similarity search with GPUscited by 2,477
33 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  3. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  4. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  5. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  6. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  7. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  8. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  9. 2019ELI5: Long Form Question Answeringcited by 567
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  11. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  12. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  13. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  14. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  15. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  16. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  17. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  18. 2016Opponent Modeling in Deep Reinforcement Learningcited by 94
  19. 2020Pre-training via Paraphrasingcited by 92
  20. 2018How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarkscited by 86
  21. 2019A Discrete Hard EM Approach for Weakly Supervised Question Answeringcited by 78
  22. 2019Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Indexcited by 64
  23. 2018What Makes Reading Comprehension Questions Easier?cited by 59
  24. 2019Unsupervised Question Answering by Cloze Translationcited by 59
  25. 2019Knowledge Guided Text Retrieval and Reading for Open Domain Question Answeringcited by 59
  26. 2019Quizbowl: The Case for Incremental Question Answeringcited by 36
  27. 2020A Memory Efficient Baseline for Open Domain Question Answeringcited by 32
  28. 2020Is Retriever Merely an Approximator of Reader?cited by 30
  29. 2019What Question Answering can Learn from Trivia Nerdscited by 21
  30. 2020Posterior Differential Regularization with f-divergence for Improving Model Robustnesscited by 20
  31. 2019Technical report on Conversational Question Answeringcited by 18
  32. 2020Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answeringcited by 17
  33. 2020Rethinking the Objectives of Extractive Question Answeringcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 19; 2023: 6; 2024: 7; 2025: 0; 2026: 2.

52 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
  3. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  4. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  5. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  6. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  7. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 110
  8. 2021Efficient Nearest Neighbor Language Modelscited by 93
  9. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  10. 2021GooAQ: Open Question Answering with Diverse Answer Typescited by 53
  11. 2021Multilingual Autoregressive Entity Linkingcited by 50
  12. 2021Semantic Answer Similarity for Evaluating Question Answering Modelscited by 47
38 more
  1. 2021One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrievalcited by 46
  2. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  3. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  4. 2022An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Taskscited by 36
  5. 2021Challenges in Generalization in Open Domain Question Answeringcited by 30
  6. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generationcited by 27
  7. 2021Highly Parallel Autoregressive Entity Linking with Discriminative Correctioncited by 21
  8. 2021What's in a Name? Answer Equivalence For Open-Domain Question Answeringcited by 21
  9. 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 21
  10. 2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 21
  11. 2022A Survey for Efficient Open Domain Question Answeringcited by 20
  12. 2022Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answeringcited by 19
  13. 2021Exploiting Reasoning Chains for Multi-hop Science Question Answeringcited by 15
  14. 2022Vote'n'Rank: Revision of Benchmarking with Social Choice Theorycited by 15
  15. 2021Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representationcited by 11
  16. 2021Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Frameworkcited by 11
  17. 2022MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languagescited by 11
  18. 2024Open-World Evaluation for Retrieving Diverse Perspectivescited by 11
  19. 2021Boosted Dense Retrievercited by 10
  20. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10
  21. 2024Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question Answeringcited by 8
  22. 2023Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluationcited by 7
  23. 2021BEAMetrics: A Benchmark for Language Generation Evaluation Evaluationcited by 6
  24. 2021Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answeringcited by 5
  25. 2024Evaluating LLMs with Multiple Problems at oncecited by 5
  26. 2022In Situ Answer Sentence Selection at Web-scalecited by 2
  27. 2022Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP modelscited by 2
  28. 2026CLEF HIPE-2026: Evaluating Accurate and Efficient Person-Place Relation Extraction from Multilingual Historical Textscited by 2
  29. 2021A Discussion on Building Practical NLP Leaderboards: The Case of Machine Translationcited by 1
  30. 2022Evaluation of Semantic Answer Similarity Metricscited by 1
  31. 2022Are We There Yet? A Decision Framework for Replacing Term Based Retrieval with Dense Retrieval Systemscited by 1
  32. 2022Improving Question Answering with Generation of NQ-like Questionscited by 1
  33. 2022Bridging the Training-Inference Gap for Dense Phrase Retrievalcited by 1
  34. 2023On Degrees of Freedom in Defining and Testing Natural Language Understandingcited by 1
  35. 2023SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative Referencescited by 1
  36. 2024Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Modelscited by 1
  37. 2022'Tis but Thy Name: Semantic Question Answering Evaluation with 11M Names for 1M Entitiescited by 0
  38. 2022Investigating the Failure Modes of the AUC metric and Exploring Alternatives for Evaluating Systems in Safety Critical Applicationscited by 0

50 most cited of 52 All 52 (JSON)