mlsanityData to 7 Oct 2026

ChEmREF: Evaluating Language Model Readiness for Chemical Emergency Response

arXiv 2511.10027 ↗First posted 13 Nov 2025cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Emergency responders managing hazardous material HAZMAT incidents face critical, time-sensitive decisions, manually navigating extensive chemical guidelines. We investigate whether today's language models can assist responders by rapidly and reliably understanding critical information, identifying hazards, and providing recommendations. We introduce the Chemical Emergency Response Evaluation Framework (ChEmREF), a new benchmark comprising questions on 1,035 HAZMAT chemicals from the Emergency Response Guidebook and the PubChem Database. ChEmREF is organized into three tasks: (1) translation of chemical representation between structured and unstructured forms (e.g., converting C2H6O to ethanol), (2) emergency response generation (e.g., recommending appropriate evacuation distances) and (3) domain knowledge question answering from chemical safety and certification exams. Our best evaluated models received an exact match of 68.0% on unstructured HAZMAT chemical representation translation, a LLM Judge score of 52.7% on incident response recommendations, and a multiple-choice accuracy of 63.9% on HAMZAT examinations. These findings suggest that while language models show potential to assist emergency responders in various tasks, they require careful human oversight due to their current limitations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 31 references that are in the map. First posted on arXiv 13 Nov 2025.

Builds on

23 of 31 references.

References · 23

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2023Large Language Modelscited by 774
  11. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  12. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
11 more
  1. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  2. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  3. 2022Translation between Molecules and Natural Languagecited by 173
  4. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 165
  5. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  6. 2023Unifying Molecular and Textual Representations via Multi-task Language Modellingcited by 92
  7. 2024Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approachescited by 51
  8. 2024From Generalist to Specialist: A Survey of Large Language Models for Chemistrycited by 11
  9. 2019Improving CNN-based Planar Object Detection with Geometric Prior Knowledgecited by 2
  10. 2020DeepHAZMAT: Hazardous Materials Sign Detection and Segmentation with Restricted Computational Resourcescited by 1
  11. 2021AI Agents in Emergency Response Applicationscited by 1

Built on by

Not cited yet.