# A Multi-modal Approach to Single-modal Visual Place Classification

arXiv: https://arxiv.org/abs/2305.06179 · HTML: https://mlsanity.com/paper/2305.06179/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 18 of 18)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [PlaNet - Photo Geolocation with Convolutional Neural Networks](https://mlsanity.com/paper/1602.05314/) (2016)
- [Structured Scene Memory for Vision-Language Navigation](https://mlsanity.com/paper/2103.03454/) (2021)
- [Deep Visual Domain Adaptation](https://mlsanity.com/paper/2012.14176/) (2020)
- [A*3D Dataset: Towards Autonomous Driving in Challenging Environments](https://mlsanity.com/paper/1909.07541/) (2019)
- [Semantic Visual Localization](https://mlsanity.com/paper/1712.05773/) (2017)
- [Auxiliary Tasks Speed Up Learning PointGoal Navigation](https://mlsanity.com/paper/2007.04561/) (2020)
- [iBoW-LCD: An Appearance-based Loop Closure Detection Approach using Incremental Bags of Binary Words](https://mlsanity.com/paper/1802.05909/) (2018)
- [ES6D: A Computation Efficient and Symmetry-Aware 6D Pose Regression Framework](https://mlsanity.com/paper/2204.01080/) (2022)
- [Are We Ready for Radar to Replace Lidar in All-Weather Mapping and Localization?](https://mlsanity.com/paper/2203.10174/) (2022)
- [Active Domain-Invariant Self-Localization Using Ego-Centric and World-Centric Maps](https://mlsanity.com/paper/2204.10497/) (2022)

## Abstract (arXiv)

Visual place classification from a first-person-view monocular RGB image is a fundamental problem in long-term robot navigation. A difficulty arises from the fact that RGB image classifiers are often vulnerable to spatial and appearance changes and degrade due to domain shifts, such as seasonal, weather, and lighting differences. To address this issue, multi-sensor fusion approaches combining RGB and depth (D) (e.g., LIDAR, radar, stereo) have gained popularity in recent years. Inspired by these efforts in multimodal RGB-D fusion, we explore the use of pseudo-depth measurements from recently-developed techniques of ``domain invariant" monocular depth estimation as an additional pseudo depth modality, by reformulating the single-modal RGB image classification task as a pseudo multi-modal RGB-D classification problem. Specifically, a practical, fully self-supervised framework for training, appropriately processing, fusing, and classifying these two modalities, RGB and pseudo-D, is described. Experiments on challenging cross-domain scenarios using public NCLT datasets validate effectiveness of the proposed framework.