# Where, What, Whether: Multi-modal Learning Meets Pedestrian Detection

arXiv: https://arxiv.org/abs/2012.10880 · HTML: https://mlsanity.com/paper/2012.10880/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2020.

## Built on by (most cited first, 6 of 6)

- [Deep Learning Serves Traffic Safety Analysis: A Forward-looking Review](https://mlsanity.com/paper/2203.10939/) (2022)
- [NMS-Loss: Learning with Non-Maximum Suppression for Crowded Pedestrian Detection](https://mlsanity.com/paper/2106.02426/) (2021)
- [View Birdification in the Crowd: Ground-Plane Localization from Perceived Movements](https://mlsanity.com/paper/2111.05060/) (2021)
- [Learning better representations for crowded pedestrians in offboard LiDAR-camera 3D tracking-by-detection](https://mlsanity.com/paper/2505.16029/) (2025)
- [Imagine the Unseen: Occluded Pedestrian Detection via Adversarial Feature Completion](https://mlsanity.com/paper/2405.01311/) (2024)
- [Towards Driver Behavior Understanding: Weakly-Supervised Risk Perception in Driving Scenes](https://mlsanity.com/paper/2603.05926/) (2026)

## Abstract (arXiv)

Pedestrian detection benefits greatly from deep convolutional neural networks (CNNs). However, it is inherently hard for CNNs to handle situations in the presence of occlusion and scale variation. In this paper, we propose W3Net, which attempts to address above challenges by decomposing the pedestrian detection task into textbftextitWhere, textbftextitWhat and textbftextitWhether problem directing against pedestrian localization, scale prediction and classification correspondingly. Specifically, for a pedestrian instance, we formulate its feature by three steps. i) We generate a bird view map, which is naturally free from occlusion issues, and scan all points on it to look for suitable locations for each pedestrian instance. ii) Instead of utilizing pre-fixed anchors, we model the interdependency between depth and scale aiming at generating depth-guided scales at different locations for better matching instances of different sizes. iii) We learn a latent vector shared by both visual and corpus space, by which false positives with similar vertical structure but lacking human partial features would be filtered out. We achieve state-of-the-art results on widely used datasets (Citypersons and Caltech). In particular. when evaluating on heavy occlusion subset, our results reduce MR-2 from 49.3% to 18.7% on Citypersons, and from 45.18% to 28.33% on Caltech.