mlsanityData to 14 Oct 2026

Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Models

arXiv 2309.15278 ↗First posted 26 Sep 2023cs.RO cs.AI cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Robots need to have a memory of previously observed, but currently occluded objects to work reliably in realistic environments. We investigate the problem of encoding object-oriented memory into a multi-object manipulation reasoning and planning framework. We propose DOOM and LOOM, which leverage transformer relational dynamics to encode the history of trajectories given partial-view point clouds and an object discovery and tracking engine. Our approaches can perform multiple challenging tasks including reasoning with occluded objects, novel objects appearance, and object reappearance. Throughout our extensive simulation and real-world experiments, we find that our approaches perform well in terms of different numbers of objects and different numbers of distractor actions. Furthermore, we show our approaches outperform an implicit memory baseline.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 47 of its 62 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

47 of 62 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  4. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  5. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  6. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 657
  7. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  8. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  9. 2019Video Object Segmentation using Space-Time Memory Networkscited by 438
  10. 2016Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Agecited by 411
  11. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  12. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 150
35 more
  1. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  2. 2022MeMOT: Multi-Object Tracking with Memorycited by 114
  3. 2019The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentationcited by 97
  4. 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
  5. 2020Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphscited by 83
  6. 2021Video Instance Segmentation with a Propose-Reduce Paradigmcited by 74
  7. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  8. 2019Online Replanning in Belief Space for Partially Observable Task and Motion Problemscited by 61
  9. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 59
  10. 2020Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Viewscited by 58
  11. 2019Towards Practical Multi-Object Manipulation using Relational Reinforcement Learningcited by 55
  12. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 50
  13. 2021On Positional and Structural Node Features for Graph Neural Networks on Non-attributed Graphscited by 49
  14. 2017Recurrent Autoregressive Networks for Online Multi-Object Trackingcited by 48
  15. 2020Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Imagecited by 48
  16. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  17. 2021Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordancescited by 40
  18. 2020Learning 3D Dynamic Scene Representations for Robot Manipulationcited by 39
  19. 2020UnOVOST: Unsupervised Offline Video Object Segmentation and Trackingcited by 34
  20. 2021NeRP: Neural Rearrangement Planning for Unknown Objectscited by 34
  21. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29
  22. 2021Predicting Stable Configurations for Semantic Placement of Novel Objectscited by 24
  23. 2022PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Trainingcited by 24
  24. 2021Efficient and Interpretable Robot Manipulation with Graph Neural Networkscited by 22
  25. 2022Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learningcited by 22
  26. 2021Target-Aware Object Discovery and Association for Unsupervised Video Multi-Object Segmentationcited by 21
  27. 2021Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulationcited by 20
  28. 2022Sequence-Based Plan Feasibility Prediction for Efficient Task and Motion Planningcited by 19
  29. 2022Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifierscited by 12
  30. 2022Memory-based gaze prediction in deep imitation learning for robot manipulationcited by 9
  31. 2020Relational Learning for Skill Preconditionscited by 8
  32. 2023Structural Concept Learning via Graph Attention for Multi-Level Rearrangement Planningcited by 8
  33. 2023Latent Space Planning for Multi-Object Manipulation with Environment-Aware Relational Classifierscited by 6
  34. 2015Monocular SLAM Supported Object Recognitioncited by 5
  35. 2023Maximal Cliques on Multi-Frame Proposal Graph for Unsupervised Video Object Segmentationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 0; 2026: 1.

5 papers.

  1. 2024Flow as the Cross-Domain Manipulation Interfacecited by 132
  2. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 67
  3. 2024Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memorycited by 11
  4. 2024Robots Can Multitask Too: Integrating a Memory Architecture and LLMs for Enhanced Cross-Task Robot Action Generationcited by 5
  5. 2026Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Taskscited by 3