# Multispectral Deep Neural Networks for Pedestrian Detection

arXiv: https://arxiv.org/abs/1611.02644 · HTML: https://mlsanity.com/paper/1611.02644/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2016.

## Built on by (most cited first, 30 of 82)

- [Cross-Modality Fusion Transformer for Multispectral Object Detection](https://mlsanity.com/paper/2111.00273/) (2021)
- [ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection](https://mlsanity.com/paper/2308.07504/) (2023)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [Fusion-Mamba for Cross-modality Object Detection](https://mlsanity.com/paper/2404.09146/) (2024)
- [C2Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection](https://mlsanity.com/paper/2306.16175/) (2023)
- [Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection](https://mlsanity.com/paper/2209.13801/) (2022)
- [Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery](https://mlsanity.com/paper/2112.02991/) (2021)
- [DAMSDet: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion](https://mlsanity.com/paper/2403.00326/) (2024)
- [Weakly Aligned Feature Fusion for Multimodal Object Detection](https://mlsanity.com/paper/2204.09848/) (2022)
- [Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving](https://mlsanity.com/paper/2105.12713/) (2021)
- [Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion](https://mlsanity.com/paper/2410.01506/) (2024)
- [DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensing](https://mlsanity.com/paper/2407.08132/) (2024)
- [Emergent Visual Sensors for Autonomous Vehicles](https://mlsanity.com/paper/2205.09383/) (2022)
- [TFDet: Target-Aware Fusion for RGB-T Pedestrian Detection](https://mlsanity.com/paper/2305.16580/) (2023)
- [DeepFusion: A Robust and Modular 3D Object Detector for Lidars, Cameras and Radars](https://mlsanity.com/paper/2209.12729/) (2022)
- [Removal then Selection: A Coarse-to-Fine Fusion Perspective for RGB-Infrared Object Detection](https://mlsanity.com/paper/2401.10731/) (2024)
- [RGB-X Object Detection via Scene-Specific Fusion Modules](https://mlsanity.com/paper/2310.19372/) (2023)
- [AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detection](https://mlsanity.com/paper/2405.12944/) (2024)
- [MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization](https://mlsanity.com/paper/2302.00290/) (2023)
- [WaveMamba: Wavelet-Driven Mamba Fusion for RGB-Infrared Object Detection](https://mlsanity.com/paper/2507.18173/) (2025)
- [Object Detection in Thermal Spectrum for Advanced Driver-Assistance Systems (ADAS)](https://mlsanity.com/paper/2109.09854/) (2021)
- [BAANet: Learning Bi-directional Adaptive Attention Gates for Multispectral Pedestrian Detection](https://mlsanity.com/paper/2112.02277/) (2021)
- [Deep Learning Serves Traffic Safety Analysis: A Forward-looking Review](https://mlsanity.com/paper/2203.10939/) (2022)
- [Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection](https://mlsanity.com/paper/2403.01300/) (2024)
- [Cross-Modality Proposal-guided Feature Mining for Unregistered RGB-Thermal Pedestrian Detection](https://mlsanity.com/paper/2308.12111/) (2023)
- [Siamese Infrared and Visible Light Fusion Network for RGB-T Tracking](https://mlsanity.com/paper/2103.07302/) (2021)
- [Self-training Guided Adversarial Domain Adaptation For Thermal Imagery](https://mlsanity.com/paper/2106.07165/) (2021)
- [Revisiting Modality Imbalance In Multimodal Pedestrian Detection](https://mlsanity.com/paper/2302.12589/) (2023)
- [When Pedestrian Detection Meets Multi-Modal Learning: Generalist Model and Benchmark Dataset](https://mlsanity.com/paper/2407.10125/) (2024)
- [Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection](https://mlsanity.com/paper/2405.16038/) (2024)

## Abstract (arXiv)

Multispectral pedestrian detection is essential for around-the-clock applications, e.g., surveillance and autonomous driving. We deeply analyze Faster R-CNN for multispectral pedestrian detection task and then model it into a convolutional network (ConvNet) fusion problem. Further, we discover that ConvNet-based pedestrian detectors trained by color or thermal images separately provide complementary information in discriminating human instances. Thus there is a large potential to improve pedestrian detection by using color and thermal images in DNNs simultaneously. We carefully design four ConvNet fusion architectures that integrate two-branch ConvNets on different DNNs stages, all of which yield better performance compared with the baseline detector. Our experimental results on KAIST pedestrian benchmark show that the Halfway Fusion model that performs fusion on the middle-level convolutional features outperforms the baseline method by 11% and yields a missing rate 3.5% lower than the other proposed architectures.