# Cross-Modality Proposal-guided Feature Mining for Unregistered RGB-Thermal Pedestrian Detection

arXiv: https://arxiv.org/abs/2308.12111 · HTML: https://mlsanity.com/paper/2308.12111/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 54 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 29 of 29)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [VarifocalNet: An IoU-aware Dense Object Detector](https://mlsanity.com/paper/2008.13367/) (2020)
- [Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems](https://mlsanity.com/paper/2008.03043/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection](https://mlsanity.com/paper/1803.05347/) (2018)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [Fusion of Multispectral Data Through Illumination-aware Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1802.09972/) (2018)
- [Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection](https://mlsanity.com/paper/2209.13801/) (2022)
- [Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1808.04818/) (2018)
- [Scale-aware Fast R-CNN for Pedestrian Detection](https://mlsanity.com/paper/1510.08160/) (2015)
- [What Can Help Pedestrian Detection?](https://mlsanity.com/paper/1705.02757/) (2017)
- [Ten Years of Pedestrian Detection, What Have We Learned?](https://mlsanity.com/paper/1411.4304/) (2014)
- [Multi-Modal Pedestrian Detection with Large Misalignment Based on Modal-Wise Regression and Multi-Modal IoU](https://mlsanity.com/paper/2107.11196/) (2021)

## Built on by (most cited first, 12 of 12)

- [Fusion-Mamba for Cross-modality Object Detection](https://mlsanity.com/paper/2404.09146/) (2024)
- [Removal then Selection: A Coarse-to-Fine Fusion Perspective for RGB-Infrared Object Detection](https://mlsanity.com/paper/2401.10731/) (2024)
- [WaveMamba: Wavelet-Driven Mamba Fusion for RGB-Infrared Object Detection](https://mlsanity.com/paper/2507.18173/) (2025)
- [Learning A Robust RGB-Thermal Detector for Extreme Modality Imbalance](https://mlsanity.com/paper/2505.22154/) (2025)
- [RGB-T Object Detection via Group Shuffled Multi-receptive Attention and Multi-modal Supervision](https://mlsanity.com/paper/2405.18955/) (2024)
- [RefHCM: A Unified Model for Referring Perceptions in Human-Centric Scenarios](https://mlsanity.com/paper/2412.14643/) (2024)
- [Person Segmentation and Action Classification for Multi-Channel Hemisphere Field of View LiDAR Sensors](https://mlsanity.com/paper/2411.11151/) (2024)
- [Modality-Decoupled RGB-Thermal Object Detector via Query Fusion](https://mlsanity.com/paper/2601.08458/) (2026)
- [LTV-YOLO: A Lightweight Thermal Object Detector for Young Pedestrians in Adverse Conditions](https://mlsanity.com/paper/2601.11662/) (2026)
- [Fusing in 3D: Free-Viewpoint Fusion Rendering with a 3D Infrared-Visible Scene Representation](https://mlsanity.com/paper/2601.12697/) (2026)
- [Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion](https://mlsanity.com/paper/2606.30215/) (2026)
- [RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection](https://mlsanity.com/paper/2608.04833/) (2026)

## Abstract (arXiv)

RGB-Thermal (RGB-T) pedestrian detection aims to locate the pedestrians in RGB-T image pairs to exploit the complementation between the two modalities for improving detection robustness in extreme conditions. Most existing algorithms assume that the RGB-T image pairs are well registered, while in the real world they are not aligned ideally due to parallax or different field-of-view of the cameras. The pedestrians in misaligned image pairs may locate at different positions in two images, which results in two challenges: 1) how to achieve inter-modality complementation using spatially misaligned RGB-T pedestrian patches, and 2) how to recognize the unpaired pedestrians at the boundary. To deal with these issues, we propose a new paradigm for unregistered RGB-T pedestrian detection, which predicts two separate pedestrian locations in the RGB and thermal images, respectively. Specifically, we propose a cross-modality proposal-guided feature mining (CPFM) mechanism to extract the two precise fusion features for representing the pedestrian in the two modalities, even if the RGB-T image pair is unaligned. It enables us to effectively exploit the complementation between the two modalities. With the CPFM mechanism, we build a two-stream dense detector; it predicts the two pedestrian locations in the two modalities based on the corresponding fusion feature mined by the CPFM mechanism. Besides, we design a data augmentation method, named Homography, to simulate the discrepancy in scales and views between images. We also investigate two non-maximum suppression (NMS) methods for post-processing. Favorable experimental results demonstrate the effectiveness and robustness of our method in dealing with unregistered pedestrians with different shifts.