# Revisiting Misalignment in Multispectral Pedestrian Detection: A Language-Driven Approach for Cross-modal Alignment Fusion

arXiv: https://arxiv.org/abs/2411.17995 · HTML: https://mlsanity.com/paper/2411.17995/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems](https://mlsanity.com/paper/2008.03043/) (2020)
- [Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection](https://mlsanity.com/paper/1803.05347/) (2018)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection](https://mlsanity.com/paper/2403.01300/) (2024)
- [Investigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning](https://mlsanity.com/paper/2005.10987/) (2020)
- [Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detection](https://mlsanity.com/paper/2311.01025/) (2023)
- [Defending Person Detection Against Adversarial Patch Attack by using Universal Defensive Frame](https://mlsanity.com/paper/2204.13004/) (2022)
- [MSCoTDet: Language-driven Multi-modal Fusion for Improved Multispectral Pedestrian Detection](https://mlsanity.com/paper/2403.15209/) (2024)
- [Robust Pedestrian Detection via Constructing Versatile Pedestrian Knowledge Bank](https://mlsanity.com/paper/2404.19299/) (2024)

## Abstract (arXiv)

Multispectral pedestrian detection is a crucial component in various critical applications. However, a significant challenge arises due to the misalignment between these modalities, particularly under real-world conditions where data often appear heavily misaligned. Conventional methods developed on well-aligned or minimally misaligned datasets fail to address these discrepancies adequately. This paper introduces a new framework for multispectral pedestrian detection designed specifically to handle heavily misaligned datasets without the need for costly and complex traditional pre-processing calibration. By leveraging Large-scale Vision-Language Models (LVLM) for cross-modal semantic alignment, our approach seeks to enhance detection accuracy by aligning semantic information across the RGB and thermal domains. This method not only simplifies the operational requirements but also extends the practical usability of multispectral detection technologies in practical applications.