# Evaluating ChatGPT on Nuclear Domain-Specific Data

arXiv: https://arxiv.org/abs/2409.00090 · HTML: https://mlsanity.com/paper/2409.00090/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 8 of 8)

- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answering](https://mlsanity.com/paper/2305.11541/) (2023)
- [LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMs](https://mlsanity.com/paper/2309.00841/) (2023)
- [Generative AI Text Classification using Ensemble LLM Approaches](https://mlsanity.com/paper/2309.07755/) (2023)
- [Practical Text Classification With Large Pre-Trained Language Models](https://mlsanity.com/paper/1812.01207/) (2018)
- [SHAP values for Explaining CNN-based Text Classification Models](https://mlsanity.com/paper/2008.11825/) (2020)

## Built on by (most cited first, 2 of 2)

- [From Naive RAG to Deep Agentic Retrieval: An Evolving Context Engineering Pipeline for Regulatory Compliance](https://mlsanity.com/paper/2607.24791/) (2026)
- [AI-Assisted Knowledge Access for Legacy Enterprise Asset Management in Energy Operations: A Practical Retrieval System](https://mlsanity.com/paper/2607.24792/) (2026)

## Abstract (arXiv)

This paper examines the application of ChatGPT, a large language model (LLM), for question-and-answer (Q&A) tasks in the highly specialized field of nuclear data. The primary focus is on evaluating ChatGPT's performance on a curated test dataset, comparing the outcomes of a standalone LLM with those generated through a Retrieval Augmented Generation (RAG) approach. LLMs, despite their recent advancements, are prone to generating incorrect or 'hallucinated' information, which is a significant limitation in applications requiring high accuracy and reliability. This study explores the potential of utilizing RAG in LLMs, a method that integrates external knowledge bases and sophisticated retrieval techniques to enhance the accuracy and relevance of generated outputs. In this context, the paper evaluates ChatGPT's ability to answer domain-specific questions, employing two methodologies: A) direct response from the LLM, and B) response from the LLM within a RAG framework. The effectiveness of these methods is assessed through a dual mechanism of human and LLM evaluation, scoring the responses for correctness and other metrics. The findings underscore the improvement in performance when incorporating a RAG pipeline in an LLM, particularly in generating more accurate and contextually appropriate responses for nuclear domain-specific queries. Additionally, the paper highlights alternative approaches to further refine and improve the quality of answers in such specialized domains.