mlsanityData to 7 Oct 2026

Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mind

arXiv 2404.05072 ↗First posted 7 Apr 2024cs.CVcited by 28

Not on the map. Why?

Abstract (arXiv)

As humans move around, performing their daily tasks, they are able to recall where they have positioned objects in their environment, even if these objects are currently out of their sight. In this paper, we aim to mimic this spatial cognition ability. We thus formulate the task of Out of Sight, Not Out of Mind - 3D tracking active objects using observations captured through an egocentric camera. We introduce a simple but effective approach to address this challenging problem, called Lift, Match, and Keep (LMK). LMK lifts partial 2D observations to 3D world coordinates, matches them over time using visual appearance, 3D location and interactions to form object tracks, and keeps these object tracks even when they go out-of-view of the camera. We benchmark LMK on 100 long videos from EPIC-KITCHENS. Our results demonstrate that spatial cognition is critical for correctly locating objects over short and long time scales. E.g., for one long egocentric video, we estimate the 3D location of 50 active objects. After 120 seconds, 57% of the objects are correctly localised by LMK, compared to just 33% by a recent 3D method for egocentric videos and 17% by a general 2D tracking method.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 36 of its 64 references that are in the map. First posted on arXiv 7 Apr 2024.

Builds on

36 of 64 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  6. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  7. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
  8. 2016Performance Measures and a Data Set for Multi-Target, Multi-Camera Trackingcited by 685
  9. 2020HOTA: A Higher Order Metric for Evaluating Multi-Object Trackingcited by 546
  10. 2020Rescaling Egocentric Visioncited by 529
  11. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  12. 2020FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Trackingcited by 489
24 more
  1. 2019Tracking without bells and whistlescited by 382
  2. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  3. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  4. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  5. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  6. 2021Anticipative Video Transformercited by 197
  7. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 166
  8. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  9. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 140
  10. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  11. 2021Learning to Track with Object Permanencecited by 117
  12. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  13. 2023TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environmentscited by 89
  14. 2023EgoHumans: An Egocentric 3D Multi-Human Benchmarkcited by 58
  15. 2023EPIC Fields: Marrying 3D Geometry and Video Understandingcited by 49
  16. 2023PACE: Human and Camera Motion Estimation from in-the-wild Videoscited by 33
  17. 2020Detecting Invisible Peoplecited by 31
  18. 2020Learning Object Permanence from Videocited by 29
  19. 2023EgoTracks: A Long-term Egocentric Visual Object Tracking Datasetcited by 27
  20. 2022EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queriescited by 25
  21. 2021Tracking People with 3D Representationscited by 22
  22. 2023Instance Tracking in 3D Scenes from Egocentric Videoscited by 17
  23. 2021Safety-aware Motion Prediction with Unseen Vehicles for Autonomous Drivingcited by 15
  24. 2023Tracking through Containers and Occluders in the Wildcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 15; 2026: 7.

28 papers.

Sort:

By area:

  1. 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 122
  2. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  3. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  4. 2024Estimating Body and Hand Motion in an Ego-sensed Worldcited by 56
  5. 2025Dynamic Point Maps: A Versatile Representation for Dynamic 3D Reconstructioncited by 43
  6. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  7. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  8. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  9. 2024Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memorycited by 11
  10. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11
  11. 2026WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videoscited by 7
  12. 2026Ego-Grounding for Personalized Question-Answering in Egocentric Videoscited by 6
16 more
  1. 20243D-Aware Instance Segmentation and Tracking in Egocentric Videoscited by 5
  2. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  3. 2025Photoreal Scene Reconstruction from an Egocentric Devicecited by 2
  4. 2025The Potential and Limitations of Vision-Language Models for Human Motion Understanding: A Case Study in Data-Driven Stroke Rehabilitationcited by 2
  5. 2025Reconstructing Objects along Hand Interaction Timelines in Egocentric Videocited by 2
  6. 2026Whareformer: Learning to Track What is Where in Long Egocentric Videoscited by 2
  7. 2025Is Tracking really more challenging in First Person Egocentric Vision?cited by 1
  8. 2025Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answeringcited by 1
  9. 2026Improving and Evaluating Hand-Object Interaction Detectioncited by 1
  10. 2025DIV-FF: Dynamic Image-Video Feature Fields For Environment Understanding in Egocentric Videoscited by 0
  11. 2025Layered Motion Fusion: Lifting Motion Segmentation to 3D in Egocentric Videoscited by 0
  12. 2025Extreme Amodal Face Detectioncited by 0
  13. 2025The N-Body Problem: Parallel Execution from Single-Person Egocentric Videocited by 0
  14. 2026G2TAM: Geometry Grounded Track Anything Modelcited by 0
  15. 2026EgoTrack3D: A Modular Framework for Egocentric 3D Object Trackingcited by 0
  16. 2026ReWorld-Track: A Recursive Event World Model for Language-Guided Multi-Camera Trackingcited by 0