InstaIndoor and Multi-modal Deep Learning for Indoor Scene Recognition
Not on the map. Why?
Abstract (arXiv)
Indoor scene recognition is a growing field with great potential for behaviour understanding, robot localization, and elderly monitoring, among others. In this study, we approach the task of scene recognition from a novel standpoint, using multi-modal learning and video data gathered from social media. The accessibility and variety of social media videos can provide realistic data for modern scene recognition techniques and applications. We propose a model based on fusion of transcribed speech to text and visual features, which is used for classification on a novel dataset of social media videos of indoor scenes named InstaIndoor. Our model achieves up to 70% accuracy and 0.7 F1-Score. Furthermore, we highlight the potential of our approach by benchmarking on a YouTube-8M subset of indoor scenes as well, where it achieves 74% accuracy and 0.74 F1-Score. We hope the contributions of this work pave the way to novel research in the challenging field of indoor scene recognition.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 84 references that are in the map. First posted on arXiv 23 Dec 2021.
Builds on
30 of 84 references.
References · 30
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
- 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
- 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,313
- 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 935
- 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
- 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
- 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
18 more
- 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
- 2019D2-Net: A Trainable CNN for Joint Detection and Description of Local Featurescited by 324
- 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
- 2021Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognitioncited by 236
- 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
- 2019An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognitioncited by 151
- 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
- 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
- 2016Places: An Image Database for Deep Scene Understandingcited by 76
- 2017From Lifestyle Vlogs to Everyday Interactionscited by 42
- 2021Generalized Contrastive Optimization of Siamese Networks for Place Recognitioncited by 27
- 2018Multimodal feature fusion for CNN-based gait recognition: an empirical comparisoncited by 14
- 2018Visual Data Synthesis via GAN for Zero-Shot Video Classificationcited by 10
- 2018From Trailers to Storylines: An Efficient Way to Learn from Moviescited by 10
- 2015A Discriminative Representation of Convolutional Features for Indoor Scene Recognitioncited by 6
- 2016Tracking Illicit Drug Dealing and Abuse on Instagram using Multimodal Analysiscited by 6
- 2015Evaluating Two-Stream CNN for Video Classificationcited by 5
- 2019Hierarchical approach to classify food scenes in egocentric photo-streamscited by 3
Built on by
Citing papers per year: 2023: 1.
1 paper.