mlsanityData to 7 Oct 2026

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

arXiv 1809.02789 ↗First posted 8 Sep 2018cs.CLcited by 2,114

Not on the map. Why?

Cited by 189 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject. The open book that comes with our questions is a set of 1329 elementary level science facts. Roughly 6000 questions probe an understanding of these facts and their application to novel situations. This requires combining an open book fact (e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of armor is made of metal) obtained from other sources. While existing QA datasets over documents or knowledge bases, being generally self-contained, focus on linguistic understanding, OpenBookQA probes a deeper understanding of both the topic---in the context of common knowledge---and the language it is expressed in. Human performance on OpenBookQA is close to 92%, but many state-of-the-art pre-trained QA methods perform surprisingly poorly, worse than several simple neural baselines we develop. Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts. We leave it as a challenge to solve the retrieval problem in this multi-hop setting and to close the large gap to human performance.

Key facts. Cited by 2,114 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 96; 2023: 218; 2024: 527; 2025: 662; 2026: 560.

2,114 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023Mistral 7Bcited by 3,065
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
38 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  8. 2023BloombergGPT: A Large Language Model for Financecited by 923
  9. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  10. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  13. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  14. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  15. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  16. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  17. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  18. 2022Galactica: A Large Language Model for Sciencecited by 736
  19. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  20. 2024Yi: Open Foundation Models by 01.AIcited by 706
  21. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  22. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  23. 2022Large Language Models Can Self-Improvecited by 656
  24. 2024Self-Rewarding Language Modelscited by 654
  25. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  26. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  27. 2023A Comprehensive Overview of Large Language Modelscited by 639
  28. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  29. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  30. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  31. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  32. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  33. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  34. 2024Scaling and evaluating sparse autoencoderscited by 530
  35. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  36. 2024Large Language Models: A Surveycited by 506
  37. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  38. 2021MetaICL: Learning to Learn In Contextcited by 450

50 most cited of 2,114 All 2,114 (JSON)