mlsanityData to 7 Oct 2026

Visual Semantic Planning using Deep Successor Representations

arXiv 1705.08080 ↗First posted 23 May 2017cs.CV cs.LG cs.ROcited by 45

Not on the map. Why?

Abstract (arXiv)

A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world. In this work, we address the problem of visual semantic planning: the task of predicting a sequence of actions from visual observations that transform a dynamic environment from an initial state to a goal state. Doing so entails knowledge about objects and their affordances, as well as actions and their preconditions and effects. We propose learning these through interacting with a visual and dynamic environment. Our proposed solution involves bootstrapping reinforcement learning with imitation learning. To ensure cross task generalization, we develop a deep predictive model based on successor representations. Our experimental results show near optimal results across a wide range of tasks in the challenging THOR environment.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 13; 2022: 11; 2023: 8; 2024: 8; 2025: 2; 2026: 2.

45 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  3. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  4. 2021Visual Room Rearrangementcited by 101
  5. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  6. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  7. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  8. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  9. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 47
  10. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  11. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  12. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environmentscited by 32
33 more
  1. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  2. 2021Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Taskscited by 25
  3. 2021Differentiable Spatial Planning using Transformerscited by 25
  4. 2022Last-Mile Embodied Visual Navigationcited by 25
  5. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  6. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  7. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  8. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  9. 2022Ask4Help: Learning to Leverage an Expert for Embodied Taskscited by 18
  10. 2023Omnidirectional Information Gathering for Knowledge Transfer-based Audio-Visual Navigationcited by 18
  11. 2021Shaping embodied agent behavior with activity-context priors from egocentric videocited by 17
  12. 2024Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challengecited by 13
  13. 2019Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigationcited by 12
  14. 2023Combining Behaviors with the Successor Features Keyboardcited by 12
  15. 2024Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examplescited by 9
  16. 2022A Rubric for Human-like Agents and NeuroAIcited by 8
  17. 2023Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hearcited by 8
  18. 2023Composing Task Knowledge with Modular Successor Feature Approximatorscited by 7
  19. 2023Multi-Level Compositional Reasoning for Interactive Instruction Followingcited by 7
  20. 2021Successor Feature Sets: Generalizing Successor Representations Across Policiescited by 6
  21. 2022Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Visioncited by 4
  22. 2022Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experiencecited by 4
  23. 2024DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Controlcited by 4
  24. 2025Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Reviewcited by 3
  25. 2021Deep Learning for Embodied Vision Navigation: A Surveycited by 2
  26. 2024Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videoscited by 2
  27. 2022ASC me to Do Anything: Multi-task Training for Embodied AIcited by 1
  28. 2022Understanding Natural Language in Contextcited by 1
  29. 2024Decoupling Exploration and Exploitation for Unsupervised Pre-training with Successor Featurescited by 1
  30. 2026Semantic Audio-Visual Navigation in Continuous Environmentscited by 1
  31. 2026SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agentscited by 1
  32. 2021VSGM -- Enhance robot task understanding ability through visual semantic graphcited by 0
  33. 2025What to Do Next? Memorizing skills from Egocentric Instructional Videocited by 0