mlsanityData to 7 Oct 2026

Semantics for Robotic Mapping, Perception and Interaction: A Survey

arXiv 2101.00443 ↗First posted 2 Jan 2021cs.RO cs.CV cs.HCcited by 56

Not on the map. Why?

Abstract (arXiv)

For robots to navigate and interact more richly with the world around them, they will likely require a deeper understanding of the world in which they operate. In robotics and related research fields, the study of understanding is often referred to as semantics, which dictates what does the world "mean" to a robot, and is strongly tied to the question of how to represent that meaning. With humans and robots increasingly operating in the same world, the prospects of human-robot interaction also bring semantics and ontology of natural language into the picture. Driven by need, as well as by enablers like increasing availability of training data and computational resources, semantics is a rapidly growing research area in robotics. The field has received significant attention in the research literature to date, but most reviews and surveys have focused on particular aspects of the topic: the technical research issues regarding its use in specific robotic topics like mapping or segmentation, or its relevance to one particular application domain like autonomous driving. A new treatment is therefore required, and is also timely because so much relevant research has occurred since many of the key surveys were published. This survey therefore provides an overarching snapshot of where semantics in robotics stands today. We establish a taxonomy for semantics research in or relevant to robotics, split into four broad categories of activity, in which semantics are extracted, used, or both. Within these broad categories we survey dozens of major topics including fundamentals from the computer vision field and key robotics research areas utilizing semantics, including mapping, navigation and interaction with the world. The survey also covers key practical considerations, including enablers like increased data availability and improved computational hardware, and major application areas where...

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 300 of its 943 references that are in the map. First posted on arXiv 2 Jan 2021.

Builds on

300 of 943 references.

References · 300

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2017Mask R-CNNcited by 7,259
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  9. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  11. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  12. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
38 more
  1. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  3. 2015Fast R-CNNcited by 2,930
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  5. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  6. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  7. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  8. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  11. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  13. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  14. 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 935
  15. 2016ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Camerascited by 801
  16. 2018Panoptic Segmentationcited by 796
  17. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  18. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  19. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  20. 2016Multi-View 3D Object Detection Network for Autonomous Drivingcited by 721
  21. 2014Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognitioncited by 719
  22. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  23. 2019Panoptic Feature Pyramid Networkscited by 644
  24. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  25. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
  26. 2019Deep Learning for 3D Point Clouds: A Surveycited by 557
  27. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  28. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  29. 2015Towards Open Set Deep Networkscited by 533
  30. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 531
  31. 2016Direct Sparse Odometrycited by 479
  32. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  33. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  34. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 455
  35. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  36. 2016R-FCN: Object Detection via Region-based Fully Convolutional Networkscited by 414
  37. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  38. 2016Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Agecited by 405

50 most cited of 300 All 300 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 12; 2023: 10; 2024: 8; 2025: 5; 2026: 7.

56 papers.

Sort:

By area:

  1. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  2. 2021Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognitioncited by 236
  3. 2021Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Imagescited by 98
  4. 2021Where is your place, Visual Place Recognition?cited by 82
  5. 20213D Semantic Scene Completion: a Surveycited by 74
  6. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  7. 2022Density-aware NeRF Ensembles: Quantifying Predictive Uncertainty in Neural Radiance Fieldscited by 58
  8. 2021SeqNet: Learning Descriptors for Sequence-based Hierarchical Place Recognitioncited by 56
  9. 2022Multi-Robot Collaborative Perception with Graph Neural Networkscited by 34
  10. 2022Behavioral Intention Prediction in Driving Scenes: A Surveycited by 30
  11. 2024SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphscited by 29
  12. 2023Semantic Image Segmentation: Two Decades of Researchcited by 26
38 more
  1. 2021Unsupervised Domain Adaptation for Semantic Image Segmentation: a Comprehensive Surveycited by 19
  2. 2024Revisit Anything: Visual Place Recognition via Image Segment Retrievalcited by 19
  3. 2022Object Permanence Emerges in a Random Walk along Memorycited by 18
  4. 2021A Hierarchical Dual Model of Environment- and Place-Specific Utility for Visual Place Recognitioncited by 15
  5. 2024GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understandingcited by 15
  6. 2022MultiRes-NetVLAD: Augmenting Place Recognition Training with Low-Resolution Imagerycited by 14
  7. 2021Conflict Avoidance in Social Navigation -- a Surveycited by 13
  8. 2021CrowdDriven: A New Challenging Dataset for Outdoor Visual Localizationcited by 13
  9. 2023Semantic Scene Completion with Cleaner Selfcited by 12
  10. 2023Navya3DSeg -- Navya 3D Semantic Segmentation Dataset & split generation for autonomous vehiclescited by 9
  11. 2021See Yourself in Others: Attending Multiple Tasks for Own Failure Detectioncited by 8
  12. 2021Hippocampal formation-inspired probabilistic generative modelcited by 5
  13. 2022Improving Worst Case Visual Localization Coverage via Place-specific Sub-selection in Multi-camera Systemscited by 5
  14. 2022Perspective-1-Ellipsoid: Formulation, Analysis and Solutions of the Camera Pose Estimation Problem from One Ellipse-Ellipsoid Correspondencecited by 5
  15. 2024QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understandingcited by 5
  16. 2021Dynamic Semantic Occupancy Mapping using 3D Scene Flow and Closed-Form Bayesian Inferencecited by 4
  17. 2022SCIM: Simultaneous Clustering, Inference, and Mapping for Open-World Semantic Scene Understandingcited by 4
  18. 2024Robot Instance Segmentation with Few Annotations for Graspingcited by 4
  19. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  20. 2022Improving Road Segmentation in Challenging Domains Using Similar Place Priorscited by 3
  21. 2023Asynchronous Events-based Panoptic Segmentation using Graph Mixer Neural Networkcited by 3
  22. 2024GSLAMOT: A Tracklet and Query Graph-based Simultaneous Locating, Mapping, and Multiple Object Tracking Systemcited by 3
  23. 2021A Deep Learning Driven Algorithmic Pipeline for Autonomous Navigation in Row-Based Cropscited by 2
  24. 2022Active Exploration based on Information Gain by Particle Filter for Efficient Spatial Concept Formationcited by 2
  25. 2023Autonomous Advanced Aerial Mobility -- An End-to-end Autonomy Framework for UAVs and Beyondcited by 2
  26. 2025EvidMTL: Evidential Multi-Task Learning for Uncertainty-Aware Semantic Surface Mapping from Monocular RGB Imagescited by 2
  27. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  28. 2026SemGS: Feed-Forward Semantic 3D Gaussian Splatting from Sparse Views for Generalizable Scene Understandingcited by 2
  29. 2022Lightweight Object-level Topological Semantic Mapping and Long-term Global Localization based on Graph Matchingcited by 1
  30. 2022Hierarchical Path-planning from Speech Instructions with Spatial Concept-based Topometric Semantic Mappingcited by 1
  31. 2023DisPlacing Objects: Improving Dynamic Vehicle Detection via Visual Place Recognition under Adverse Conditionscited by 1
  32. 2023CurriculumLoc: Enhancing Cross-Domain Geolocalization through Multi-Stage Refinementcited by 1
  33. 2025LLMs Can Also Do Well! Breaking Barriers in Semantic Role Labeling via Large Language Modelscited by 1
  34. 2026Modernising Reinforcement Learning-Based Navigation for Embodied Semantic Scene Graph Generationcited by 1
  35. 2021SeqNetVLAD vs PointNetVLAD: Image Sequence vs 3D Point Clouds for Day-Night Place Recognitioncited by 0
  36. 2023PENet: A Joint Panoptic Edge Detection Networkcited by 0
  37. 2023Locking On: Leveraging Dynamic Vehicle-Imposed Motion Constraints to Improve Visual Localizationcited by 0
  38. 2024MORTAR: A Model-based Runtime Action Repair Framework for AI-enabled Cyber-Physical Systemscited by 0

50 most cited of 56 All 56 (JSON)