# NuclearQA: A Human-Made Benchmark for Language Models for the Nuclear Domain

arXiv: https://arxiv.org/abs/2310.10920 · HTML: https://mlsanity.com/paper/2310.10920/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 29 of 29)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [TheoremQA: A Theorem-driven Question Answering dataset](https://mlsanity.com/paper/2305.12524/) (2023)
- [emrQA: A Large Corpus for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1809.00732/) (2018)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge](https://mlsanity.com/paper/2102.03315/) (2021)
- [BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing](https://mlsanity.com/paper/2206.15076/) (2022)
- [BIOMRC: A Dataset for Biomedical Machine Reading Comprehension](https://mlsanity.com/paper/2005.06376/) (2020)
- [Towards an Atlas of Cultural Commonsense for Machine Reasoning](https://mlsanity.com/paper/2009.05664/) (2020)
- [NukeBERT: A Pre-trained language model for Low Resource Nuclear Domain](https://mlsanity.com/paper/2003.13821/) (2020)
- [OneStop QAMaker: Extract Question-Answer Pairs from Text in a One-Stop Approach](https://mlsanity.com/paper/2102.12128/) (2021)

## Built on by (most cited first, 7 of 7)

- [MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science](https://mlsanity.com/paper/2510.12171/) (2025)
- [A Survey of Pre-trained Language Models for Processing Scientific Text](https://mlsanity.com/paper/2401.17824/) (2024)
- [Post Turing: Mapping the landscape of LLM Evaluation](https://mlsanity.com/paper/2311.02049/) (2023)
- [WeQA: A Benchmark for Retrieval Augmented Generation in Wind Energy Domain](https://mlsanity.com/paper/2408.11800/) (2024)
- [The Trust Calibration Maturity Model for Characterizing and Communicating Trustworthiness of AI Systems](https://mlsanity.com/paper/2503.15511/) (2025)
- [NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models](https://mlsanity.com/paper/2606.27047/) (2026)
- [LLM-Guided Planning for Multi-hop Reasoning over Multimodal Nuclear Regulatory Documents](https://mlsanity.com/paper/2606.29399/) (2026)

## Abstract (arXiv)

As LLMs have become increasingly popular, they have been used in almost every field. But as the application for LLMs expands from generic fields to narrow, focused science domains, there exists an ever-increasing gap in ways to evaluate their efficacy in those fields. For the benchmarks that do exist, a lot of them focus on questions that don't require proper understanding of the subject in question. In this paper, we present NuclearQA, a human-made benchmark of 100 questions to evaluate language models in the nuclear domain, consisting of a varying collection of questions that have been specifically designed by experts to test the abilities of language models. We detail our approach and show how the mix of several types of questions makes our benchmark uniquely capable of evaluating models in the nuclear domain. We also present our own evaluation metric for assessing LLM's performances due to the limitations of existing ones. Our experiments on state-of-the-art models suggest that even the best LLMs perform less than satisfactorily on our benchmark, demonstrating the scientific knowledge gap of existing LLMs.