# Semantic-Enhanced Cross-Modal Place Recognition for Robust Robot Localization

arXiv: https://arxiv.org/abs/2509.13474 · HTML: https://mlsanity.com/paper/2509.13474/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Builds on (most cited first, 30 of 30)

- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://mlsanity.com/paper/2101.11038/) (2021)
- [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [Template-Based Named Entity Recognition Using BART](https://mlsanity.com/paper/2106.01760/) (2021)
- [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024)
- [Visual News: Benchmark and Challenges in News Image Captioning](https://mlsanity.com/paper/2010.03743/) (2020)
- [ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension](https://mlsanity.com/paper/2204.05991/) (2022)
- [Probing Image-Language Transformers for Verb Understanding](https://mlsanity.com/paper/2106.09141/) (2021)
- [E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning](https://mlsanity.com/paper/2106.01804/) (2021)
- [Cross-modal Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2205.02444/) (2022)
- [On Pursuit of Designing Multi-modal Transformer for Video Grounding](https://mlsanity.com/paper/2109.06085/) (2021)
- [Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback](https://mlsanity.com/paper/2501.01377/) (2025)
- [Meta-Learning for Domain Generalization in Semantic Parsing](https://mlsanity.com/paper/2010.11988/) (2020)
- [Multimodal End-to-End Sparse Model for Emotion Recognition](https://mlsanity.com/paper/2103.09666/) (2021)
- [KITTI-CARLA: a KITTI-like dataset generated by CARLA Simulator](https://mlsanity.com/paper/2109.00892/) (2021)
- [Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2109.02401/) (2021)
- [LAMBDA: A Large Model Based Data Agent](https://mlsanity.com/paper/2407.17535/) (2024)
- [Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation](https://mlsanity.com/paper/2505.24787/) (2025)
- [N-LTP: An Open-source Neural Language Technology Platform for Chinese](https://mlsanity.com/paper/2009.11616/) (2020)
- [Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2310.05804/) (2023)
- [ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition](https://mlsanity.com/paper/2112.06482/) (2021)
- [A Survey on Large Language Model-based Agents for Statistics and Data Science](https://mlsanity.com/paper/2412.14222/) (2024)
- [CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection](https://mlsanity.com/paper/2204.05515/) (2022)
- [Multiplex Graph Neural Network for Extractive Text Summarization](https://mlsanity.com/paper/2108.12870/) (2021)
- [LlamaCare: A Large Medical Language Model for Enhancing Healthcare Knowledge Sharing](https://mlsanity.com/paper/2406.02350/) (2024)
- [Adaptive Field Effect Planner for Safe Interactive Autonomous Driving on Curved Roads](https://mlsanity.com/paper/2504.14747/) (2025)
- [Data-Driven Evolutionary Game-Based Model Predictive Control for Hybrid Renewable Energy Dispatch in Autonomous Ships](https://mlsanity.com/paper/2504.14750/) (2025)

## Abstract (arXiv)

Ensuring accurate localization of robots in environments without GPS capability is a challenging task. Visual Place Recognition (VPR) techniques can potentially achieve this goal, but existing RGB-based methods are sensitive to changes in illumination, weather, and other seasonal changes. Existing cross-modal localization methods leverage the geometric properties of RGB images and 3D LiDAR maps to reduce the sensitivity issues highlighted above. Currently, state-of-the-art methods struggle in complex scenes, fine-grained or high-resolution matching, and situations where changes can occur in viewpoint. In this work, we introduce a framework we call Semantic-Enhanced Cross-Modal Place Recognition (SCM-PR) that combines high-level semantics utilizing RGB images for robust localization in LiDAR maps. Our proposed method introduces: a VMamba backbone for feature extraction of RGB images; a Semantic-Aware Feature Fusion (SAFF) module for using both place descriptors and segmentation masks; LiDAR descriptors that incorporate both semantics and geometry; and a cross-modal semantic attention mechanism in NetVLAD to improve matching. Incorporating the semantic information also was instrumental in designing a Multi-View Semantic-Geometric Matching and a Semantic Consistency Loss, both in a contrastive learning framework. Our experimental work on the KITTI and KITTI-360 datasets show that SCM-PR achieves state-of-the-art performance compared to other cross-modal place recognition methods.