mlsanityData to 7 Oct 2026

RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation

arXiv 2402.15487 ↗First posted 23 Feb 2024cs.RO cs.AI cs.CVcited by 48

Not on the map. Why?

Abstract (arXiv)

We introduce the novel task of interactive scene exploration, wherein robots autonomously explore environments and produce an action-conditioned scene graph (ACSG) that captures the structure of the underlying environment. The ACSG accounts for both low-level information (geometry and semantics) and high-level information (action-conditioned relationships between different entities) in the scene. To this end, we present the Robotic Exploration (RoboEXP) system, which incorporates the Large Multimodal Model (LMM) and an explicit memory design to enhance our system's capabilities. The robot reasons about what and how to explore an object, accumulating new information through the interaction process and incrementally constructing the ACSG. Leveraging the constructed ACSG, we illustrate the effectiveness and efficiency of our RoboEXP system in facilitating a wide range of real-world manipulation tasks involving rigid, articulated objects, nested objects, and deformable objects.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 95 of its 134 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

95 of 134 references.

References · 95

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  12. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
38 more
  1. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  2. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  3. 2023Segment Anything in High Qualitycited by 373
  4. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  5. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  6. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  7. 2017Embodied Question Answeringcited by 335
  8. 2016Visual Relationship Detection with Language Priorscited by 333
  9. 2018Image Generation from Scene Graphscited by 324
  10. 2020Learning to Explore using Active Neural SLAMcited by 308
  11. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  12. 2018Graph R-CNN for Scene Graph Generationcited by 296
  13. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  14. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  15. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  16. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  17. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  18. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  19. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  20. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  21. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  22. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  23. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  24. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  25. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  26. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  27. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  28. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  29. 2018Visual Semantic Navigation using Scene Priorscited by 127
  30. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  31. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  32. 2017Visual Translation Embedding Network for Visual Relation Detectioncited by 112
  33. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  34. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  35. 2021Visual Room Rearrangementcited by 101
  36. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  37. 2018Visual Representations for Semantic Target Driven Navigationcited by 89
  38. 2019Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environmentscited by 89

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 25; 2026: 17.

48 papers.

Sort:

By area:

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  2. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  3. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 41
  4. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  5. 2025Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flowcited by 21
  6. 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interactioncited by 14
  7. 2025Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Modelscited by 13
  8. 2024The Bare Necessities: Designing Simple, Effective Open-Vocabulary Scene Graphscited by 11
  9. 2024FLAIR: Feeding via Long-horizon AcquIsition of Realistic dishescited by 10
  10. 2025WoMAP: World Models For Embodied Open-Vocabulary Object Localizationcited by 8
  11. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguitycited by 8
  12. 2025MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planningcited by 8
36 more
  1. 2025ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Modelscited by 7
  2. 2026Towards the Harness of Embodied Agentscited by 7
  3. 2024BiKC: Keypose-Conditioned Consistency Policy for Bimanual Robotic Manipulationcited by 5
  4. 2025KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulationcited by 5
  5. 2025Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Modelscited by 5
  6. 2025CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graphcited by 4
  7. 2025Robot Operation of Home Appliances by Reading User Manualscited by 4
  8. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  9. 2025Human-like Navigation in a World Built for Humanscited by 4
  10. 2026Towards Exploratory and Focused Manipulation with Bimanual Active Perception: A New Problem, Benchmark and Strategycited by 4
  11. 2026FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flowcited by 4
  12. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  13. 2026Overcoming Statistical Bias in Action-Controllable World Modelscited by 4
  14. 2024EMPOWER: Embodied Multi-role Open-vocabulary Planning with Online Grounding and Executioncited by 3
  15. 2025Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulationcited by 3
  16. 2026Pandora: Articulated 3D Scene Graphs from Egocentric Visioncited by 3
  17. 2024Learning to Look: Seeking Information for Decision Making via Policy Factorizationcited by 2
  18. 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Modelscited by 2
  19. 2025Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmarkcited by 2
  20. 2025Hierarchical DLO Routing with Reinforcement Learning and In-Context Vision-language Modelscited by 2
  21. 2025Generalizable Hierarchical Skill Learning via Object-Centric Representationcited by 2
  22. 2026LOST-3DSG: Lightweight Open-Vocabulary 3D Scene Graphs with Semantic Tracking in Dynamic Environmentscited by 2
  23. 2025SG-Tailor: Inter-Object Commonsense Relationship Reasoning for Scene Graph Manipulationcited by 1
  24. 2025MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioningcited by 1
  25. 2026VLM-DEWM: Dynamic External World Model for Verifiable and Resilient Vision-Language Planning in Manufacturingcited by 1
  26. 2026EvoMemNav: Efficient Self-Evolving Fine-Grained Memory for Zero-Shot Embodied Navigationcited by 1
  27. 2025Preference-Driven Active 3D Scene Representation for Robotic Inspection in Nuclear Decommissioningcited by 0
  28. 2025Proactive Scene Decomposition and Reconstructioncited by 0
  29. 2026CoReLIN: Constraint-based Reasoning for Zero-shot Lifelong Interactive Navigationcited by 0
  30. 2026Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learningcited by 0
  31. 2026Fixed External Cameras as Common Prior Maps for Active 3D Scene Graph Generationcited by 0
  32. 2026RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robotscited by 0
  33. 2026DGSG-Mind: Dynamic 3D Gaussian Scene Graphs for Long-Term Scene Understanding and Groundingcited by 0
  34. 2026Expanding Spatial and Temporal Context for Robotic Imitation Learning With Scene Graphscited by 0
  35. 2026Active Spatial Inspection for Effective and Efficient Embodied Explorationcited by 0
  36. 2026DynaHarness: A Dynamic Physical Harness for Self-Evolving Robot Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RoboEvolve: Co-Evolving Planner-Simulator for Robotic Manipulation with Limited Data
  2. 2026From Passive Execution to Active Exploration: Agentic Embodied Manipulation in Realistic Environments
  3. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
  4. 2025RoboEngine: Plug-and-Play Robot Data Augmentation with Semantic Robot Segmentation and Background Generation
  5. 2024Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulation
  6. 2025AdaManip: Adaptive Articulated Object Manipulation Environments and Policy Learning
  7. 2026RoboQuest: Generalist Physical Agents that Search, Inspect and Test
  8. 2026EmbodimentSemantic: A Spatial Scene-Graph Dataset and Benchmark for Vision-Language Models on Embodied Manipulation Trajectories