mlsanityData to 7 Oct 2026

VSGM -- Enhance robot task understanding ability through visual semantic graph

arXiv 2105.08959 ↗First posted 19 May 2021cs.RO cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In recent years, developing AI for robotics has raised much attention. The interaction of vision and language of robots is particularly difficult. We consider that giving robots an understanding of visual semantics and language semantics will improve inference ability. In this paper, we propose a novel method-VSGM (Visual Semantic Graph Memory), which uses the semantic graph to obtain better visual image features, improve the robot's visual understanding ability. By providing prior knowledge of the robot and detecting the objects in the image, it predicts the correlation between the attributes of the object and the objects and converts them into a graph-based representation; and mapping the object in the image to be a top-down egocentric map. Finally, the important object features of the current task are extracted by Graph Neural Networks. The method proposed in this paper is verified in the ALFRED (Action Learning From Realistic Environments and Directives) dataset. In this dataset, the robot needs to perform daily indoor household tasks following the required language instructions. After the model is added to the VSGM, the task success rate can be improved by 6~10%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 66 references that are in the map. First posted on arXiv 19 May 2021.

Builds on

48 of 66 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Graph Attention Networkscited by 8,854
  4. 2017Mask R-CNNcited by 7,259
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
36 more
  1. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  2. 2019Graph Transformer Networkscited by 520
  3. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  4. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  5. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  6. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  7. 2020Learning to Explore using Active Neural SLAMcited by 308
  8. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  9. 2018Graph R-CNN for Scene Graph Generationcited by 296
  10. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  11. 2019Vision-and-Dialog Navigationcited by 246
  12. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  13. 2020Neural Topological SLAM for Visual Navigationcited by 193
  14. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  15. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
  16. 2018Visual Semantic Navigation using Scene Priorscited by 127
  17. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  18. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  19. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  20. 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
  21. 2018Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learningcited by 91
  22. 2020Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphscited by 83
  23. 2018Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generationcited by 78
  24. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  25. 2020Learning Object Relation Graph and Tentative Policy for Visual Navigationcited by 62
  26. 2020Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Viewscited by 57
  27. 2020A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Taskscited by 52
  28. 2017Visual Semantic Planning using Deep Successor Representationscited by 45
  29. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  30. 2019Regression Planning Networkscited by 32
  31. 2019SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigationcited by 29
  32. 2020Learning to plan with uncertain topological mapscited by 29
  33. 2021MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigationcited by 11
  34. 2021Are We There Yet? Learning to Localize in Embodied Instruction Followingcited by 7
  35. 2021A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environmentcited by 6
  36. 2021Spatial Imagination With Semantic Cognition for Mobile Robotscited by 3

Built on by

Not cited yet.