# Multi-modal Visual Place Recognition in Dynamics-Invariant Perception Space

arXiv: https://arxiv.org/abs/2105.07800 · HTML: https://mlsanity.com/paper/2105.07800/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 17 May 2021.

## Builds on (most cited first, 21 of 21)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [Lending Orientation to Neural Networks for Cross-view Geo-localization](https://mlsanity.com/paper/1903.12351/) (2019)
- [Optimal Feature Transport for Cross-View Image Geo-Localization](https://mlsanity.com/paper/1907.05021/) (2019)
- [Semantic Visual Localization](https://mlsanity.com/paper/1712.05773/) (2017)
- [Beyond Controlled Environments: 3D Camera Re-Localization in Changing Indoor Scenes](https://mlsanity.com/paper/2008.02004/) (2020)
- [See Clearer at Night: Towards Robust Nighttime Semantic Segmentation through Day-Night Image Conversion](https://mlsanity.com/paper/1908.05868/) (2019)
- [Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN](https://mlsanity.com/paper/1706.03319/) (2017)
- [Robust Neural Routing Through Space Partitions for Camera Relocalization in Dynamic Indoor Environments](https://mlsanity.com/paper/2012.04746/) (2020)
- [Learning Matchable Image Transformations for Long-term Metric Visual Localization](https://mlsanity.com/paper/1904.01080/) (2019)
- [Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space](https://mlsanity.com/paper/1809.10239/) (2018)
- [Empty Cities: a Dynamic-Object-Invariant Space for Visual SLAM](https://mlsanity.com/paper/2010.07646/) (2020)

## Built on by (most cited first, 1 of 1)

- [Framework Construction of an Adversarial Federated Transfer Learning Classifier](https://mlsanity.com/paper/2211.04734/) (2022)

## Abstract (arXiv)

Visual place recognition is one of the essential and challenging problems in the fields of robotics. In this letter, we for the first time explore the use of multi-modal fusion of semantic and visual modalities in dynamics-invariant space to improve place recognition in dynamic environments. We achieve this by first designing a novel deep learning architecture to generate the static semantic segmentation and recover the static image directly from the corresponding dynamic image. We then innovatively leverage the spatial-pyramid-matching model to encode the static semantic segmentation into feature vectors. In parallel, the static image is encoded using the popular Bag-of-words model. On the basis of the above multi-modal features, we finally measure the similarity between the query image and target landmark by the joint similarity of their semantic and visual codes. Extensive experiments demonstrate the effectiveness and robustness of the proposed approach for place recognition in dynamic environments.