# Learning Cross-Modal Deep Representations for Robust Pedestrian Detection

arXiv: https://arxiv.org/abs/1704.02431 · HTML: https://mlsanity.com/paper/1704.02431/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2017.

## Built on by (most cited first, 30 of 34)

- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection](https://mlsanity.com/paper/2209.13801/) (2022)
- [Cross-modal Consensus Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2107.12589/) (2021)
- [Cross-Spectral Neural Radiance Fields](https://mlsanity.com/paper/2209.00648/) (2022)
- [Weakly Aligned Feature Fusion for Multimodal Object Detection](https://mlsanity.com/paper/2204.09848/) (2022)
- [Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving](https://mlsanity.com/paper/2105.12713/) (2021)
- [UniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning](https://mlsanity.com/paper/2404.17360/) (2024)
- [AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detection](https://mlsanity.com/paper/2405.12944/) (2024)
- [MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization](https://mlsanity.com/paper/2302.00290/) (2023)
- [Robust 3D Object Detection from LiDAR-Radar Point Clouds via Cross-Modal Feature Augmentation](https://mlsanity.com/paper/2309.17336/) (2023)
- [Siamese Infrared and Visible Light Fusion Network for RGB-T Tracking](https://mlsanity.com/paper/2103.07302/) (2021)
- [Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens](https://mlsanity.com/paper/2401.01674/) (2024)
- [Probabilistic Graph Attention Network with Conditional Kernels for Pixel-Wise Prediction](https://mlsanity.com/paper/2101.02843/) (2021)
- [Pedestrian Detection in Low-Light Conditions: A Comprehensive Survey](https://mlsanity.com/paper/2401.07801/) (2024)
- [Cross-spectral Gated-RGB Stereo Depth Estimation](https://mlsanity.com/paper/2405.12759/) (2024)
- [Robust pedestrian detection in thermal imagery using synthesized images](https://mlsanity.com/paper/2102.02005/) (2021)
- [Multi-Modal Pedestrian Detection with Large Misalignment Based on Modal-Wise Regression and Multi-Modal IoU](https://mlsanity.com/paper/2107.11196/) (2021)
- [Pseudo-LiDAR Based Road Detection](https://mlsanity.com/paper/2107.13279/) (2021)
- [MTMMC: A Large-Scale Real-World Multi-Modal Camera Tracking Benchmark](https://mlsanity.com/paper/2403.20225/) (2024)
- [A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospects](https://mlsanity.com/paper/2506.19769/) (2025)
- [Surformer v1: Transformer-Based Surface Classification Using Tactile and Vision Features](https://mlsanity.com/paper/2508.06566/) (2025)
- [A Survey on Deep Domain Adaptation and Tiny Object Detection Challenges, Techniques and Datasets](https://mlsanity.com/paper/2107.07927/) (2021)
- [Illumination and Temperature-Aware Multispectral Networks for Edge-Computing-Enabled Pedestrian Detection](https://mlsanity.com/paper/2112.05053/) (2021)
- [Cascaded information enhancement and cross-modal attention feature fusion for multispectral pedestrian detection](https://mlsanity.com/paper/2302.08670/) (2023)
- [Erasure-based Interaction Network for RGBT Video Object Detection and A Unified Benchmark](https://mlsanity.com/paper/2308.01630/) (2023)
- [MS23D: A 3D Object Detection Method Using Multi-Scale Semantic Feature Points to Construct 3D Feature Layer](https://mlsanity.com/paper/2308.16518/) (2023)
- [MSCoTDet: Language-driven Multi-modal Fusion for Improved Multispectral Pedestrian Detection](https://mlsanity.com/paper/2403.15209/) (2024)
- [Variational Structured Attention Networks for Deep Visual Representation Learning](https://mlsanity.com/paper/2103.03510/) (2021)
- [Partially fake it till you make it: mixing real and fake thermal images for improved object detection](https://mlsanity.com/paper/2106.13603/) (2021)

## Abstract (arXiv)

This paper presents a novel method for detecting pedestrians under adverse illumination conditions. Our approach relies on a novel cross-modality learning framework and it is based on two main phases. First, given a multimodal dataset, a deep convolutional network is employed to learn a non-linear mapping, modeling the relations between RGB and thermal data. Then, the learned feature representations are transferred to a second deep network, which receives as input an RGB image and outputs the detection results. In this way, features which are both discriminative and robust to bad illumination conditions are learned. Importantly, at test time, only the second pipeline is considered and no thermal data are required. Our extensive evaluation demonstrates that the proposed approach outperforms the state-of- the-art on the challenging KAIST multispectral pedestrian dataset and it is competitive with previous methods on the popular Caltech dataset.