# ASK: Adaptively Selecting Key Local Features for RGB-D Scene Recognition

arXiv: https://arxiv.org/abs/2110.07703 · HTML: https://mlsanity.com/paper/2110.07703/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning](https://mlsanity.com/paper/1807.03146/) (2018)
- [Multi-scale Orderless Pooling of Deep Convolutional Activation Features](https://mlsanity.com/paper/1403.1840/) (2014)
- [Weakly Supervised Adversarial Domain Adaptation for Semantic Segmentation in Urban Scenes](https://mlsanity.com/paper/1904.09092/) (2019)
- [Translate-to-Recognize Networks for RGB-D Scene Recognition](https://mlsanity.com/paper/1904.12254/) (2019)
- [Understand Scene Categories by Objects: A Semantic Regularized Scene Classifier Using Convolutional Neural Networks](https://mlsanity.com/paper/1509.06470/) (2015)
- [Learning Effective RGB-D Representations for Scene Recognition](https://mlsanity.com/paper/1809.06269/) (2018)
- [Depth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNs](https://mlsanity.com/paper/1801.06797/) (2018)
- [Deep Regionlets for Object Detection](https://mlsanity.com/paper/1712.02408/) (2017)
- [VSSA-NET: Vertical Spatial Sequence Attention Network for Traffic Sign Detection](https://mlsanity.com/paper/1905.01583/) (2019)
- [Fisher Kernel for Deep Neural Activations](https://mlsanity.com/paper/1412.1628/) (2014)

## Built on by (most cited first, 12 of 12)

- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data](https://mlsanity.com/paper/2205.03147/) (2022)
- [Decoupling Common and Unique Representations for Multimodal Self-supervised Learning](https://mlsanity.com/paper/2309.05300/) (2023)
- [Change Detection Meets Visual Question Answering](https://mlsanity.com/paper/2112.06343/) (2021)
- [CGiS-Net: Aggregating Colour, Geometry and Implicit Semantic Features for Indoor Place Recognition](https://mlsanity.com/paper/2202.02070/) (2022)
- [CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasets](https://mlsanity.com/paper/2302.06148/) (2023)
- [Attentional Graph Convolutional Network for Structure-aware Audio-Visual Scene Classification](https://mlsanity.com/paper/2301.00145/) (2022)
- [GAMUS: A Geometry-aware Multi-modal Semantic Segmentation Benchmark for Remote Sensing Data](https://mlsanity.com/paper/2305.14914/) (2023)
- [A Deep Learning-based Global and Segmentation-based Semantic Feature Fusion Approach for Indoor Scene Classification](https://mlsanity.com/paper/2302.06432/) (2023)
- [Robust Multimodal Learning for Ophthalmic Disease Grading via Disentangled Representation](https://mlsanity.com/paper/2503.05319/) (2025)
- [Exploiting Object-based and Segmentation-based Semantic Features for Deep Learning-based Indoor Scene Classification](https://mlsanity.com/paper/2404.07739/) (2024)
- [Dynamic Graph Neural Network with Adaptive Features Selection for RGB-D Based Indoor Scene Recognition](https://mlsanity.com/paper/2604.00372/) (2026)

## Abstract (arXiv)

Indoor scene images usually contain scattered objects and various scene layouts, which make RGB-D scene classification a challenging task. Existing methods still have limitations for classifying scene images with great spatial variability. Thus, how to extract local patch-level features effectively using only image labels is still an open problem for RGB-D scene recognition. In this paper, we propose an efficient framework for RGB-D scene recognition, which adaptively selects important local features to capture the great spatial variability of scene images. Specifically, we design a differentiable local feature selection (DLFS) module, which can extract the appropriate number of key local scenerelated features. Discriminative local theme-level and object-level representations can be selected with the DLFS module from the spatially-correlated multi-modal RGB-D features. We take advantage of the correlation between RGB and depth modalities to provide more cues for selecting local features. To ensure that discriminative local features are selected, the variational mutual information maximization loss is proposed. Additionally, the DLFS module can be easily extended to select local features of different scales. By concatenating the local-orderless and global structured multi-modal features, the proposed framework can achieve state-of-the-art performance on public RGB-D scene recognition datasets.