# Psych-Occlusion: Using Visual Psychophysics for Aerial Detection of Occluded Persons during Search and Rescue

arXiv: https://arxiv.org/abs/2412.05553 · HTML: https://mlsanity.com/paper/2412.05553/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 41 of its 105 references that are in the map.
- First posted on arXiv 7 Dec 2024.

## Builds on (most cited first, 30 of 41)

- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [The Limits and Potentials of Deep Learning for Robotics](https://mlsanity.com/paper/1804.06557/) (2018)
- [PP-YOLOE: An evolved version of YOLO](https://mlsanity.com/paper/2203.16250/) (2022)
- [PROB: Probabilistic Objectness for Open World Object Detection](https://mlsanity.com/paper/2212.01424/) (2022)
- [Unmanned Aerial Vehicles: A Survey on Civil Applications and Key Research Challenges](https://mlsanity.com/paper/1805.00881/) (2018)
- [Towards Open-Set Object Detection and Discovery](https://mlsanity.com/paper/2204.05604/) (2022)
- [UC-OWOD: Unknown-Classified Open World Object Detection](https://mlsanity.com/paper/2207.11455/) (2022)
- [CYBORG: Blending Human Saliency Into the Loss Improves Deep Learning](https://mlsanity.com/paper/2112.00686/) (2021)
- [Human-Aided Saliency Maps Improve Generalization of Deep Learning](https://mlsanity.com/paper/2105.03492/) (2021)
- [Occlusion Handling in Generic Object Detection: A Review](https://mlsanity.com/paper/2101.08845/) (2021)
- [Domain-Specific Human-Inspired Binarized Statistical Image Features for Iris Recognition](https://mlsanity.com/paper/1807.05248/) (2018)
- [The State of Aerial Surveillance: A Survey](https://mlsanity.com/paper/2201.03080/) (2022)
- [Visual Psychophysics for Making Face Recognition Algorithms More Explainable](https://mlsanity.com/paper/1803.07140/) (2018)
- [PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition](https://mlsanity.com/paper/1611.06448/) (2016)
- [Robustness of Object Recognition under Extreme Occlusion in Humans and Computational Models](https://mlsanity.com/paper/1905.04598/) (2019)
- [Using Human Brain Activity to Guide Machine Learning](https://mlsanity.com/paper/1703.05463/) (2017)
- [YOLO-ReT: Towards High Accuracy Real-time Object Detection on Edge GPUs](https://mlsanity.com/paper/2110.13713/) (2021)
- [Performance of Humans in Iris Recognition: The Impact of Iris Condition and Annotation-driven Verification](https://mlsanity.com/paper/1807.05245/) (2018)
- [Measuring Human Perception to Improve Handwritten Document Transcription](https://mlsanity.com/paper/1904.03734/) (2019)
- [Human Saliency-Driven Patch-based Matching for Interpretable Post-mortem Iris Recognition](https://mlsanity.com/paper/2208.03138/) (2022)

## Abstract (arXiv)

The success of Emergency Response (ER) scenarios, such as search and rescue, is often dependent upon the prompt location of a lost or injured person. With the increasing use of small Unmanned Aerial Systems (sUAS) as "eyes in the sky" during ER scenarios, efficient detection of persons from aerial views plays a crucial role in achieving a successful mission outcome. Fatigue of human operators during prolonged ER missions, coupled with limited human resources, highlights the need for sUAS equipped with Computer Vision (CV) capabilities to aid in finding the person from aerial views. However, the performance of CV models onboard sUAS substantially degrades under real-life rigorous conditions of a typical ER scenario, where person search is hampered by occlusion and low target resolution. To address these challenges, we extracted images from the NOMAD dataset and performed a crowdsource experiment to collect behavioural measurements when humans were asked to "find the person in the picture". We exemplify the use of our behavioral dataset, Psych-ER, by using its human accuracy data to adapt the loss function of a detection model. We tested our loss adaptation on a RetinaNet model evaluated on NOMAD against increasing distance and occlusion, with our psychophysical loss adaptation showing improvements over the baseline at higher distances across different levels of occlusion, without degrading performance at closer distances. To the best of our knowledge, our work is the first human-guided approach to address the location task of a detection model, while addressing real-world challenges of aerial search and rescue. All datasets and code can be found at: https://github.com/ArtRuss/NOMAD.