mlsanityData to 7 Oct 2026

Efficient Multi-Task RGB-D Scene Analysis for Indoor Environments

arXiv 2207.04526 ↗First posted 10 Jul 2022cs.CV cs.LG cs.ROcited by 32

Not on the map. Why?

Abstract (arXiv)

Semantic scene understanding is essential for mobile agents acting in various environments. Although semantic segmentation already provides a lot of information, details about individual objects as well as the general scene are missing but required for many real-world applications. However, solving multiple tasks separately is expensive and cannot be accomplished in real time given limited computing and battery capabilities on a mobile platform. In this paper, we propose an efficient multi-task approach for RGB-D scene analysis~(EMSANet) that simultaneously performs semantic and instance segmentation~(panoptic segmentation), instance orientation estimation, and scene classification. We show that all tasks can be accomplished using a single neural network in real time on a mobile platform without diminishing performance - by contrast, the individual tasks are able to benefit from each other. In order to evaluate our multi-task approach, we extend the annotations of the common RGB-D indoor datasets NYUv2 and SUNRGB-D for instance segmentation and orientation estimation. To the best of our knowledge, we are the first to provide results in such a comprehensive multi-task setting for indoor scene analysis on NYUv2 and SUNRGB-D.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 10 Jul 2022.

Builds on

43 of 55 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2017Mask R-CNNcited by 7,259
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2017Squeeze-and-Excitation Networkscited by 4,763
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  12. 2016Pyramid Scene Parsing Networkcited by 2,472
31 more
  1. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  4. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  5. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  6. 2018Panoptic Segmentationcited by 796
  7. 2019Panoptic Feature Pyramid Networkscited by 644
  8. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  9. 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 552
  10. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  11. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  12. 2019Panoptic-DeepLabcited by 362
  13. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  14. 20163D Bounding Box Estimation Using Deep Learning and Geometrycited by 277
  15. 2021Objects are Different: Flexible Monocular 3D Object Detectioncited by 184
  16. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  17. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  18. 2020EfficientPS: Efficient Panoptic Segmentationcited by 111
  19. 2019ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentationcited by 105
  20. 2019In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Imagescited by 99
  21. 2019SSAP: Single-Shot Instance Segmentation With Affinity Pyramidcited by 85
  22. 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 83
  23. 2021ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentationcited by 75
  24. 2018Self-Supervised Model Adaptation for Multimodal Semantic Segmentationcited by 73
  25. 2019Deep Fitting Degree Scoring Network for Monocular 3D Object Detectioncited by 67
  26. 2019DeeperLab: Single-Shot Image Parsercited by 63
  27. 2021OmniDet: Surround View Cameras based Multi-task Visual Perception Network for Autonomous Drivingcited by 57
  28. 2019Real-Time Panoptic Segmentation from Dense Detectionscited by 31
  29. 2019Fast Panoptic Segmentation Networkcited by 19
  30. 2020EPSNet: Efficient Panoptic Segmentation Network with Cross-layer Attention Fusioncited by 6
  31. 2022Efficient and Robust Semantic Mapping for Indoor Environmentscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 6; 2025: 9; 2026: 7.

32 papers.

Sort:

By area:

  1. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 81
  2. 2024Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentationcited by 71
  3. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 58
  4. 2024StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentationcited by 38
  5. 2025DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentationcited by 30
  6. 2023AsymFormer: Asymmetrical Cross-Modal Representation Learning for Mobile Platform Real-Time RGB-D Semantic Segmentationcited by 16
  7. 2023Efficient Multi-Task Scene Analysis with RGB-D Transformerscited by 7
  8. 2024Perception Matters: Enhancing Embodied AI with Uncertainty-Aware Semantic Segmentationcited by 6
  9. 2023PanopticNDT: Efficient and Robust Panoptic Mappingcited by 5
  10. 2023Blocks2World: Controlling Realistic Scenes with Editable Primitivescited by 4
  11. 2023Spatial-information Guided Adaptive Context-aware Network for Efficient RGB-D Semantic Segmentationcited by 4
  12. 2025OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentationcited by 4
20 more
  1. 2023ComPtr: Towards Diverse Bi-source Dense Prediction Tasks via A Simple yet General Complementary Transformercited by 3
  2. 2025LIDAR: Lightweight Adaptive Cue-Aware Fusion Vision Mamba for Multimodal Segmentation of Structural Crackscited by 3
  3. 2025MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learningcited by 3
  4. 2025M2H: Multi-Task Learning with Efficient Window-Based Cross-Task Attention for Monocular Spatial Perceptioncited by 3
  5. 2024Depth-guided Texture Diffusion for Image Semantic Segmentationcited by 2
  6. 2025PDDM: Pseudo Depth Diffusion Model for RGB-PD Semantic Segmentation Based in Complex Indoor Scenescited by 2
  7. 2025DiffPixelFormer: Differential Pixel-Aware Transformer for RGB-D Indoor Scene Segmentationcited by 2
  8. 2023A Deep Learning-based Global and Segmentation-based Semantic Feature Fusion Approach for Indoor Scene Classificationcited by 1
  9. 2026Efficient Prediction of Dense Visual Embeddings via Distillation and RGB-D Transformerscited by 1
  10. 2026GeomPrompt: Geometric Prompt Learning for RGB-D Semantic Segmentation Under Missing and Degraded Depthcited by 1
  11. 2026RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodingscited by 1
  12. 2023Optimizing rgb-d semantic segmentation through multi-modal interaction and pooling attentioncited by 0
  13. 2024Exploiting Object-based and Segmentation-based Semantic Features for Deep Learning-based Indoor Scene Classificationcited by 0
  14. 2024Depth-aware Panoptic Segmentationcited by 0
  15. 2025AMMNet: An Asymmetric Multi-Modal Network for Remote Sensing Semantic Segmentationcited by 0
  16. 2025Power of Boundary and Reflection: Semantic Transparent Object Segmentation using Pyramid Vision Transformer with Transparent Cuescited by 0
  17. 2026Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidancecited by 0
  18. 2026CrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentationcited by 0
  19. 2026Compass: Degradation-Simulated Reciprocal Learning with Lightweight Needle RWKV for Multimodal Crack Segmentation under Missing Modalitiescited by 0
  20. 2026URNet: A Unified Reparameterized Network for Efficient RGB-D Semantic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Towards Multimodal Multitask Scene Understanding Models for Indoor Mobile Agents
  2. 2024Open-RGBT: Open-vocabulary RGB-T Zero-shot Semantic Segmentation in Open-world Environments
  3. 2026Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments
  4. 2026ReScene4D: Temporally Consistent Semantic Instance Segmentation of Evolving Indoor 3D Scenes
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environments
  6. 2026SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
  7. 2026SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Models
  8. 2025Open-Vocabulary Indoor Object Grounding with 3D Hierarchical Scene Graph