mlsanityData to 14 Oct 2026

ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis

arXiv 2504.06553 ↗First posted 9 Apr 2025cs.RO cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

While recent work in scene reconstruction and understanding has made strides in grounding natural language to physical 3D environments, it is still challenging to ground abstract, high-level instructions to a 3D scene. High-level instructions might not explicitly invoke semantic elements in the scene, and even the process of breaking a high-level task into a set of more concrete subtasks, a process called hierarchical task analysis, is environment-dependent. In this work, we propose ASHiTA, the first framework that generates a task hierarchy grounded to a 3D scene graph by breaking down high-level tasks into grounded subtasks. ASHiTA alternates LLM-assisted hierarchical task analysis, to generate the task breakdown, with task-driven 3D scene graph construction to generate a suitable representation of the environment. Our experiments show that ASHiTA performs significantly better than LLM baselines in breaking down high-level tasks into environment-dependent subtasks and is additionally able to achieve grounding performance comparable to state-of-the-art methods.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 9 Apr 2025.

Builds on

36 of 48 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Segment Anythingcited by 8,234
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  6. 2023LightGlue: Local Feature Matching at Light Speedcited by 590
  7. 2023LERF: Language Embedded Radiance Fieldscited by 543
  8. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  9. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  10. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 328
  11. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  12. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
24 more
  1. 2020Integrated Task and Motion Planningcited by 290
  2. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  3. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 205
  4. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  5. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  6. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  7. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  8. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
  9. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  10. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  11. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 135
  12. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 113
  13. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  14. 2024Clio: Real-time Task-Driven Open-Set 3D Scene Graphscited by 81
  15. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  16. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  17. 2024RTAB-Map as an Open-Source Lidar and Visual SLAM Library for Large-Scale and Long-Term Online Operationcited by 62
  18. 2023Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-time Systemscited by 60
  19. 2023How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontierscited by 54
  20. 2024Guiding Long-Horizon Task and Motion Planning with Vision Language Modelscited by 47
  21. 2024Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splattingcited by 38
  22. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  23. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 20
  24. 2024Task and Motion Planning in Hierarchical 3D Scene Graphscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Describe Anything Anywhere At Any Momentcited by 19
  2. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 10
  3. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  4. 2026STaR: Scalable Task-Conditioned Retrieval for Long-Horizon Multimodal Robot Memorycited by 0
  5. 2026Expanding Spatial and Temporal Context for Robotic Imitation Learning With Scene Graphscited by 0
  6. 2026Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environmentscited by 0
  7. 2026GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenescited by 0
  8. 2026Finder: Agentic Closed-Loop Object Finding for Embodied Groundingcited by 0