# VLPD: Context-Aware Pedestrian Detection via Vision-Language Semantic Self-Supervision

arXiv: https://arxiv.org/abs/2304.03135 · HTML: https://mlsanity.com/paper/2304.03135/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 6 Apr 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [CityPersons: A Diverse Dataset for Pedestrian Detection](https://mlsanity.com/paper/1702.05693/) (2017)
- [Rethinking Semantic Segmentation: A Prototype View](https://mlsanity.com/paper/2203.15102/) (2022)
- [Adaptive NMS: Refining Pedestrian Detection in a Crowd](https://mlsanity.com/paper/1904.03629/) (2019)
- [Repulsion Loss: Detecting Pedestrians in a Crowd](https://mlsanity.com/paper/1711.07752/) (2017)
- [Occlusion-aware R-CNN: Detecting Pedestrians in a Crowd](https://mlsanity.com/paper/1807.08407/) (2018)
- [WiderPerson: A Diverse Dataset for Dense Pedestrian Detection in the Wild](https://mlsanity.com/paper/1909.12118/) (2019)
- [NMS by Representative Region: Towards Crowded Pedestrian Detection by Proposal Pairing](https://mlsanity.com/paper/2003.12729/) (2020)
- [Progressive End-to-End Object Detection in Crowded Scenes](https://mlsanity.com/paper/2203.07669/) (2022)
- [PedHunter: Occlusion Robust Pedestrian Detector in Crowded Scenes](https://mlsanity.com/paper/1909.06826/) (2019)
- [How Far are We from Solving Pedestrian Detection?](https://mlsanity.com/paper/1602.01237/) (2016)
- [Relational Learning for Joint Head and Human Detection](https://mlsanity.com/paper/1909.10674/) (2019)
- [Pedestrian Detection with Autoregressive Network Phases](https://mlsanity.com/paper/1812.00440/) (2018)
- [NOH-NMS: Improving Pedestrian Detection by Nearby Objects Hallucination](https://mlsanity.com/paper/2007.13376/) (2020)
- [Semantic Head Enhanced Pedestrian Detection in a Crowd](https://mlsanity.com/paper/1911.11985/) (2019)
- [NMS-Loss: Learning with Non-Maximum Suppression for Crowded Pedestrian Detection](https://mlsanity.com/paper/2106.02426/) (2021)

## Built on by (most cited first, 19 of 19)

- [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023)
- [A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges](https://mlsanity.com/paper/2501.02189/) (2025)
- [DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving](https://mlsanity.com/paper/2401.03641/) (2024)
- [Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives](https://mlsanity.com/paper/2402.02968/) (2024)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detection](https://mlsanity.com/paper/2311.01025/) (2023)
- [Unsupervised Multi-view Pedestrian Detection](https://mlsanity.com/paper/2305.12457/) (2023)
- [Evaluating and Enhancing Trustworthiness of LLMs in Perception Tasks](https://mlsanity.com/paper/2408.01433/) (2024)
- [CoLA: Conditional Dropout and Language-driven Robust Dual-modal Salient Object Detection](https://mlsanity.com/paper/2407.06780/) (2024)
- [Multi-Granularity Language-Guided Training for Multi-Object Tracking](https://mlsanity.com/paper/2406.04844/) (2024)
- [Towards General Urban Monitoring with Vision-Language Models: A Review, Evaluation, and a Research Agenda](https://mlsanity.com/paper/2510.12400/) (2025)
- [Imagine the Unseen: Occluded Pedestrian Detection via Adversarial Feature Completion](https://mlsanity.com/paper/2405.01311/) (2024)
- [A Hierarchical Test Platform for Vision Language Model (VLM)-Integrated Real-World Autonomous Driving](https://mlsanity.com/paper/2506.14100/) (2025)
- [Ranking-based Adaptive Query Generation for DETRs in Crowded Pedestrian Detection](https://mlsanity.com/paper/2310.15725/) (2023)
- [Video-Language Alignment via Spatio-Temporal Graph Transformer](https://mlsanity.com/paper/2407.11677/) (2024)
- [From Camera-Based Sensing to Reasoning: A Comprehensive Review Toward Proactive Vulnerable Road User Safety](https://mlsanity.com/paper/2510.03314/) (2025)
- [From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving](https://mlsanity.com/paper/2603.17714/) (2026)
- [Visual Prompt Based Reasoning for Offroad Mapping using Multimodal LLMs](https://mlsanity.com/paper/2604.04564/) (2026)
- [MTF-Net: Multi-Modal Temporal Feature Fusion Network for Pedestrian Intention Prediction](https://mlsanity.com/paper/2609.20178/) (2026)

## Abstract (arXiv)

Detecting pedestrians accurately in urban scenes is significant for realistic applications like autonomous driving or video surveillance. However, confusing human-like objects often lead to wrong detections, and small scale or heavily occluded pedestrians are easily missed due to their unusual appearances. To address these challenges, only object regions are inadequate, thus how to fully utilize more explicit and semantic contexts becomes a key problem. Meanwhile, previous context-aware pedestrian detectors either only learn latent contexts with visual clues, or need laborious annotations to obtain explicit and semantic contexts. Therefore, we propose in this paper a novel approach via Vision-Language semantic self-supervision for context-aware Pedestrian Detection (VLPD) to model explicitly semantic contexts without any extra annotations. Firstly, we propose a self-supervised Vision-Language Semantic (VLS) segmentation method, which learns both fully-supervised pedestrian detection and contextual segmentation via self-generated explicit labels of semantic classes by vision-language models. Furthermore, a self-supervised Prototypical Semantic Contrastive (PSC) learning method is proposed to better discriminate pedestrians and other classes, based on more explicit and semantic contexts obtained from VLS. Extensive experiments on popular benchmarks show that our proposed VLPD achieves superior performances over the previous state-of-the-arts, particularly under challenging circumstances like small scale and heavy occlusion. Code is available at https://github.com/lmy98129/VLPD.