mlsanityData to 7 Oct 2026

FIction: 4D Future Interaction Prediction from Video

arXiv 2412.00932 ↗First posted 1 Dec 2024cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Anticipating how a person will interact with objects in an environment is essential for activity understanding, but existing methods are limited to the 2D space of video frames-capturing physically ungrounded predictions of "what" and ignoring the "where" and "how". We introduce FIction for 4D future interaction prediction from videos. Given an input video of a human activity, the goal is to predict which objects at what 3D locations the person will interact with in the next time period (e.g., cabinet, fridge), and how they will execute that interaction (e.g., poses for bending, reaching, pulling). Our novel model FIction fuses the past video observation of the person's actions and their environment to predict both the "where" and "how" of future interactions. Through comprehensive experiments on a variety of activities and real-world environments in EgoExo4D, we show that our proposed approach outperforms prior autoregressive and (lifted) 2D video models substantially, with more than 30% relative gains.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 96 of its 119 references that are in the map. First posted on arXiv 1 Dec 2024.

Builds on

96 of 119 references.

References · 96

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2018SlowFast Networks for Video Recognitioncited by 1,862
  7. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  8. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  12. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
38 more
  1. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  2. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  3. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 673
  4. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  5. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  6. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  7. 2020Rescaling Egocentric Visioncited by 529
  8. 2018On Evaluation of Embodied Navigation Agentscited by 526
  9. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  10. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  11. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  12. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  13. 2017VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimatorcited by 418
  14. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  15. 2017On human motion prediction using recurrent neural networkscited by 336
  16. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  17. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 306
  18. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  19. 2023OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Predictioncited by 287
  20. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  21. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  22. 2022Egocentric Video-Language Pretrainingcited by 231
  23. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  24. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  25. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  26. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  27. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  28. 2023Project Aria: A New Tool for Egocentric Multi-Modal AI Researchcited by 201
  29. 2023WHAM: Reconstructing World-grounded Humans with Accurate 3D Motioncited by 200
  30. 2021Anticipative Video Transformercited by 197
  31. 2020Long-term Human Motion Prediction with Scene Contextcited by 191
  32. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  33. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
  34. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  35. 2017SEGCloud: Semantic Segmentation of 3D Point Cloudscited by 178
  36. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  37. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  38. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 140

50 most cited of 96 All 96 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Learning Predictive Visuomotor Coordinationcited by 2
  2. 2025Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Spancited by 2
  3. 2026EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinkingcited by 1
  4. 2025Seeing My Future: Predicting Situated Interaction Behavior in Virtual Realitycited by 0
  5. 2026ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priorscited by 0
  6. 2026Ego-Human Motion Prediction with 3D-Aware LLMcited by 0
  7. 2026From Where to How: Continuous 4D Interaction Forecasting from Egocentric Videocited by 0
  8. 2026Ego-Exo4D Human Meshes Dataset: 4D Human Motion Reconstruction for Ego-Exo Capturescited by 0