# A Multimodal Manufacturing Safety Chatbot: Knowledge Base Design, Benchmark Development, and Evaluation of Multiple RAG Approaches

arXiv: https://arxiv.org/abs/2511.11847 · HTML: https://mlsanity.com/paper/2511.11847/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 29 references that are in the map.
- First posted on arXiv 14 Nov 2025.

## Builds on (most cited first, 1 of 1)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)

## Built on by (most cited first, 2 of 2)

- [A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challenges](https://mlsanity.com/paper/2602.05883/) (2026)
- [The Token Tax of Epistemic Accuracy: Comparing RAG and Long-Context Architectures for Document-Grounded Generative AI Applications](https://mlsanity.com/paper/2606.20898/) (2026)

## Abstract (arXiv)

Ensuring worker safety remains a critical challenge in modern manufacturing environments. Industry 5.0 reorients the prevailing manufacturing paradigm toward more human-centric operations. Using a design science research methodology, we identify three essential requirements for next-generation safety training systems: high accuracy, low latency, and low cost. We introduce a multimodal chatbot powered by large language models that meets these design requirements. The chatbot uses retrieval-augmented generation to ground its responses in curated regulatory and technical documentation. To evaluate our solution, we developed a domain-specific benchmark of expert-validated question and answer pairs for three representative machines: a Bridgeport manual mill, a Haas TL-1 CNC lathe, and a Universal Robots UR5e collaborative robot. We tested 24 RAG configurations using a full-factorial design and assessed them with automated evaluations of correctness, latency, and cost. Our top 2 configurations were then evaluated by ten industry experts and academic researchers. Our results show that retrieval strategy and model configuration have a significant impact on performance. The top configuration, selected for chatbot deployment, achieved an accuracy of 86.66%, an average cost of 0.005 per query, and an average end-to-end latency of 10.04 seconds. This latency is practical for delivering a complete safety instruction and is measured from query submission to full instruction delivery rather than generation onset. Overall, our work provides three contributions: an open-source, domain-grounded safety training chatbot; a validated benchmark for evaluating AI-assisted safety instruction; and a systematic methodology for designing and assessing AI-enabled instructional and immersive safety training systems for Industry 5.0 environments.