Spatially Prompted Visual Trajectory Prediction for Egocentric Manipulation
Not on the map. Why?
Abstract (arXiv)
Language instructions offer a flexible way to specify task semantics but can become ambiguous in cluttered scenes containing multiple visually similar objects and candidate targets. We study Spatially Prompted Visual Trajectory Prediction (SP-VTP), an open-loop setting in which points or bounding boxes in the first egocentric frame specify an object--target pair and condition the prediction of future 3D end-effector (EE) trajectory chunks. To support this setting, we introduce EgoSPT, a controlled dataset comprising 2,841 egocentric fork pick-and-place episodes with first-frame object and target annotations and reconstructed 3D EE trajectories. SP-VTP evaluates whether a static task specification remains informative as the viewpoint and scene configuration evolve throughout execution. We also introduce SPOT, a spatially prompted object--target trajectory predictor that integrates first-frame visual and coordinate prompts with current visual observations and motion history to generate future trajectories. Rather than relying on an explicit object tracker, SPOT maintains a fixed representation of the first-frame task specification. Across three training seeds, spatial prompting consistently reduces translation and endpoint errors relative to a no-prompt baseline, while controlled prompt interventions reveal distinct roles for object and target cues at different stages of execution. Together, EgoSPT and SPOT provide a dataset and baseline for spatially conditioned egocentric trajectory prediction in open-loop pick-and-place settings.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 59 of its 63 references that are in the map. First posted on arXiv 19 May 2026.
Builds on
59 of 63 references.
References · 59
- 2017Attention Is All You Needcited by 60,361
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2023Segment Anythingcited by 8,234
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
- 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
- 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
38 more
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
- 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
- 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
- 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
- 2020Rescaling Egocentric Visioncited by 540
- 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
- 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
- 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
- 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 346
- 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
- 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
- 2018Visual Reinforcement Learning with Imagined Goalscited by 251
- 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 236
- 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
- 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 175
- 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 145
- 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
- 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
- 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 137
- 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 133
- 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
- 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 129
- 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 111
- 2025Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfercited by 87
- 2024UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllerscited by 75
- 2025DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulationcited by 66
- 2024FastUMI: A Scalable and Hardware-Independent Universal Manipulation Interface with Datasetcited by 40
- 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
- 2025DexWild: Dexterous Human Interactions for In-the-Wild Robot Policiescited by 38
- 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 34
- 2025MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulationcited by 34
- 2022Egocentric Prediction of Action Target in 3Dcited by 31
- 2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 31
- 2023EgoTracks: A Long-term Egocentric Visual Object Tracking Datasetcited by 27
- 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 26
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.