mlsanityData to 7 Oct 2026

Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration

arXiv 1807.03480 ↗First posted 10 Jul 2018cs.CV cs.AI cs.LGcited by 60

Not on the map. Why?

Abstract (arXiv)

Our goal is to generate a policy to complete an unseen task given just a single video demonstration of the task in a given domain. We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model. To this end, we propose Neural Task Graph (NTG) Networks, which use conjugate task graph as the intermediate representation to modularize both the video demonstration and the derived policy. We empirically show NTG achieves inter-task generalization on two complex tasks: Block Stacking in BulletPhysics and Object Collection in AI2-THOR. NTG improves data efficiency with visual input as well as achieve strong generalization without the need for dense hierarchical supervision. We further show that similar performance trends hold when applied to real-world data. We show that NTG can effectively predict task structure on the JIGSAWS surgical dataset and generalize to unseen tasks.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 3; 2020: 1; 2021: 14; 2022: 16; 2023: 14; 2024: 5; 2025: 4; 2026: 2.

60 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  3. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  4. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  5. 2024OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completioncited by 78
  6. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  7. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  8. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 64
  9. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  10. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  11. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  12. 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 48
38 more
  1. 2021NeRP: Neural Rearrangement Planning for Unknown Objectscited by 33
  2. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  3. 2018One-Shot Hierarchical Imitation Learning of Compound Visuomotor Taskscited by 32
  4. 2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 32
  5. 2019Few-Shot Bayesian Imitation Learning with Logical Program Policiescited by 30
  6. 2021Predicting Stable Configurations for Semantic Placement of Novel Objectscited by 23
  7. 2022Policy Architectures for Compositional Generalization in Controlcited by 23
  8. 2023Universal Visual Decomposer: Long-Horizon Manipulation Made Easycited by 23
  9. 2020Scalable Multi-Task Imitation Learning with Autonomous Improvementcited by 21
  10. 2021Efficient and Interpretable Robot Manipulation with Graph Neural Networkscited by 21
  11. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  12. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
  13. 2022STAP: Sequencing Task-Agnostic Policiescited by 17
  14. 2022Cross-Domain Transfer via Semantic Skill Imitationcited by 17
  15. 2025Learning Compositional Behaviors from Demonstration and Languagecited by 15
  16. 2023Multimodal Subtask Graph Generation from Instructional Videoscited by 13
  17. 2023United We Stand, Divided We Fall: UnityGraph for Unsupervised Procedure Learning from Videoscited by 13
  18. 2021Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Taskscited by 12
  19. 2023EC^2: Emergent Communication for Embodied Controlcited by 10
  20. 2019Continuous Relaxation of Symbolic Planner for One-Shot Imitation Learningcited by 8
  21. 2023Action Dynamics Task Graphs for Learning Plannable Representations of Procedural Taskscited by 8
  22. 2024PRIME: Scaffolding Manipulation Tasks with Behavior Primitives for Data-Efficient Imitation Learningcited by 8
  23. 2023Masked Diffusion with Task-awareness for Procedure Planning in Instructional Videoscited by 7
  24. 2019Motion Reasoning for Goal-Based Imitation Learningcited by 6
  25. 2021TANGO: Commonsense Generalization in Predicting Tool Interactions for Mobile Manipulatorscited by 6
  26. 2022Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrationscited by 6
  27. 2023Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangementcited by 6
  28. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Modelscited by 6
  29. 2021Discovering Generalizable Skills via Automated Generation of Diverse Taskscited by 5
  30. 2022Zero-Shot Policy Transfer with Disentangled Task Representation of Meta-Reinforcement Learningcited by 5
  31. 2021Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabelingcited by 4
  32. 2023TOD-Flow: Modeling the Structure of Task-Oriented Dialoguescited by 4
  33. 2022Learning Parameterized Task Structure for Generalization to Unseen Entitiescited by 3
  34. 2022Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalizationcited by 3
  35. 2023Moving Forward by Moving Backward: Embedding Action Impact over Action Semanticscited by 3
  36. 2023Evaluating Robustness of Visual Representations for Object Assembly Task Requiring Spatio-Geometrical Reasoningcited by 3
  37. 2024Learning to Recover from Plan Execution Errors during Robot Manipulation: A Neuro-symbolic Approachcited by 3
  38. 2021Projection: A Mechanism for Human-like Reasoning in Artificial Intelligencecited by 2

50 most cited of 60 All 60 (JSON)