# Integrating Deep Learning and Augmented Reality to Enhance Situational Awareness in Firefighting Environments

arXiv: https://arxiv.org/abs/2107.11043 · HTML: https://mlsanity.com/paper/2107.11043/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 190 references that are in the map.
- First posted on arXiv 23 Jul 2021.

## Builds on (most cited first, 13 of 13)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Streaming Graph Neural Networks](https://mlsanity.com/paper/1810.10627/) (2018)
- [FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation](https://mlsanity.com/paper/1903.11816/) (2019)
- [Deep Reinforcement learning for real autonomous mobile robot navigation in indoor environments](https://mlsanity.com/paper/2005.13857/) (2020)
- [PVANET: Deep but Lightweight Neural Networks for Real-time Object Detection](https://mlsanity.com/paper/1608.08021/) (2016)
- [Distributed Non-Negative Tensor Train Decomposition](https://mlsanity.com/paper/2008.01340/) (2020)
- [MAT: A Multimodal Attentive Translator for Image Captioning](https://mlsanity.com/paper/1702.05658/) (2017)
- [Deep Neural Networks for Pattern Recognition](https://mlsanity.com/paper/1809.09645/) (2018)
- [A Novel Indoor Positioning System for unprepared firefighting scenarios](https://mlsanity.com/paper/2008.01344/) (2020)
- [An embedded deep learning system for augmented reality in firefighting applications](https://mlsanity.com/paper/2009.10679/) (2020)

## Abstract (arXiv)

We present a new four-pronged approach to build firefighter's situational awareness for the first time in the literature. We construct a series of deep learning frameworks built on top of one another to enhance the safety, efficiency, and successful completion of rescue missions conducted by firefighters in emergency first response settings. First, we used a deep Convolutional Neural Network (CNN) system to classify and identify objects of interest from thermal imagery in real-time. Next, we extended this CNN framework for object detection, tracking, segmentation with a Mask RCNN framework, and scene description with a multimodal natural language processing(NLP) framework. Third, we built a deep Q-learning-based agent, immune to stress-induced disorientation and anxiety, capable of making clear navigation decisions based on the observed and stored facts in live-fire environments. Finally, we used a low computational unsupervised learning technique called tensor decomposition to perform meaningful feature extraction for anomaly detection in real-time. With these ad-hoc deep learning structures, we built the artificial intelligence system's backbone for firefighters' situational awareness. To bring the designed system into usage by firefighters, we designed a physical structure where the processed results are used as inputs in the creation of an augmented reality capable of advising firefighters of their location and key features around them, which are vital to the rescue operation at hand, as well as a path planning feature that acts as a virtual guide to assist disoriented first responders in getting back to safety. When combined, these four approaches present a novel approach to information understanding, transfer, and synthesis that could dramatically improve firefighter response and efficacy and reduce life loss.