# A BERT Baseline for the Natural Questions

arXiv: https://arxiv.org/abs/1901.08634 · HTML: https://mlsanity.com/paper/1901.08634/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2019.

## Built on by (most cited first, 30 of 33)

- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Understanding by Understanding Not: Modeling Negation in Language Models](https://mlsanity.com/paper/2105.03519/) (2021)
- [Poolingformer: Long Document Modeling with Pooling Attention](https://mlsanity.com/paper/2105.04371/) (2021)
- [Knowing More About Questions Can Help: Improving Calibration in Question Answering](https://mlsanity.com/paper/2106.01494/) (2021)
- [Can NLI Models Verify QA Systems' Predictions?](https://mlsanity.com/paper/2104.08731/) (2021)
- [SD-QA: Spoken Dialectal Question Answering for the Real World](https://mlsanity.com/paper/2109.12072/) (2021)
- [EntQA: Entity Linking as Question Answering](https://mlsanity.com/paper/2110.02369/) (2021)
- [Capturing Row and Column Semantics in Transformer Based Question Answering over Tables](https://mlsanity.com/paper/2104.08303/) (2021)
- [Representations for Question Answering from Documents with Tables and Text](https://mlsanity.com/paper/2101.10573/) (2021)
- [What's in a Name? Answer Equivalence For Open-Domain Question Answering](https://mlsanity.com/paper/2109.05289/) (2021)
- [Reasoning over Public and Private Data in Retrieval-Based Systems](https://mlsanity.com/paper/2203.11027/) (2022)
- [MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding](https://mlsanity.com/paper/2112.10728/) (2021)
- [On Measuring Social Biases in Prompt-Based Multi-Task Learning](https://mlsanity.com/paper/2205.11605/) (2022)
- [Towards More Equitable Question Answering Systems: How Much More Data Do You Need?](https://mlsanity.com/paper/2105.14115/) (2021)
- [Automatic Dataset Generation for Knowledge Intensive Question Answering Tasks](https://mlsanity.com/paper/2505.14212/) (2025)
- [Synthetic Target Domain Supervision for Open Retrieval QA](https://mlsanity.com/paper/2204.09248/) (2022)
- [Comprehensive Overview of Named Entity Recognition: Models, Domain-Specific Applications and Challenges](https://mlsanity.com/paper/2309.14084/) (2023)
- [Toward Deconfounding the Influence of Entity Demographics for Question Answering Accuracy](https://mlsanity.com/paper/2104.07571/) (2021)
- [Do Answers to Boolean Questions Need Explanations? Yes](https://mlsanity.com/paper/2112.07772/) (2021)
- [Evaluation of Question Answering Systems: Complexity of judging a natural language](https://mlsanity.com/paper/2209.12617/) (2022)
- [Human Latency Conversational Turns for Spoken Avatar Systems](https://mlsanity.com/paper/2404.16053/) (2024)
- [PrimeQA: The Prime Repository for State-of-the-Art Multilingual Question Answering Research and Development](https://mlsanity.com/paper/2301.09715/) (2023)
- [Towards Confident Machine Reading Comprehension](https://mlsanity.com/paper/2101.07942/) (2021)
- [Investigating Post-pretraining Representation Alignment for Cross-Lingual Question Answering](https://mlsanity.com/paper/2109.12028/) (2021)
- [Analysis of the Reasoning with Redundant Information Provided Ability of Large Language Models](https://mlsanity.com/paper/2310.04039/) (2023)
- [When to Fold'em: How to answer Unanswerable questions](https://mlsanity.com/paper/2105.00328/) (2021)
- [GAAMA 2.0: An Integrated System that Answers Boolean and Extractive Questions](https://mlsanity.com/paper/2206.08441/) (2022)
- [BERT-based knowledge extraction method of unstructured domain text](https://mlsanity.com/paper/2103.00728/) (2021)
- [VAULT: VAriable Unified Long Text Representation for Machine Reading Comprehension](https://mlsanity.com/paper/2105.03229/) (2021)

## Abstract (arXiv)

This technical note describes a new baseline for the Natural Questions. Our model is based on BERT and reduces the gap between the model F1 scores reported in the original dataset paper and the human upper bound by 30% and 50% relative for the long and short answer tasks respectively. This baseline has been submitted to the official NQ leaderboard at ai.google.com/research/NaturalQuestions. Code, preprocessed data and pretrained model are available at https://github.com/google-research/language/tree/master/language/question_answering/bert_joint.