mlsanityData to 7 Oct 2026

Understanding 3D Object Interaction from a Single Image

arXiv 2305.09664 ↗First posted 16 May 2023cs.CVcited by 33

Not on the map. Why?

Abstract (arXiv)

Humans can easily understand a single image as depicting multiple potential objects permitting interaction. We use this skill to plan our interactions with the world and accelerate understanding new objects without engaging in interaction. In this paper, we would like to endow machines with the similar ability, so that intelligent agents can better explore the 3D scene or manipulate objects. Our approach is a transformer-based model that predicts the 3D location, physical properties and affordance of objects. To power this model, we collect a dataset with Internet videos, egocentric videos and indoor images to train and validate our approach. Our model yields strong performance on our data, and generalizes well to robotics data. Project site: https://jasonqsy.github.io/3DOI/

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 71 of its 81 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

71 of 81 references.

References · 71

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2023Segment Anythingcited by 8,158
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2017Mask R-CNNcited by 7,259
  10. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  11. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  12. 2021Vision Transformers for Dense Predictioncited by 1,789
38 more
  1. 2019Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regressioncited by 1,361
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  3. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  4. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  5. 2018Panoptic Segmentationcited by 796
  6. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  7. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  8. 2017Focal Loss for Dense Object Detectioncited by 749
  9. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  10. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  11. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  12. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  13. 2020Rescaling Egocentric Visioncited by 529
  14. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  15. 2019PointRend: Image Segmentation as Renderingcited by 439
  16. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  17. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  18. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 219
  19. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 214
  20. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  21. 2016What do different evaluation metrics tell us about saliency models?cited by 184
  22. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  23. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  24. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 159
  25. 2019Category-Level Articulated Object Pose Estimationcited by 158
  26. 2022Human-to-Robot Imitation in the Wildcited by 157
  27. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  28. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  29. 2021A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representationcited by 113
  30. 2018PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Imagecited by 105
  31. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  32. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  33. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 78
  34. 2018PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Imagecited by 77
  35. 2023LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Groundingcited by 64
  36. 2014Designing Deep Networks for Surface Normal Estimationcited by 59
  37. 2021AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactionscited by 56
  38. 2022OPD: Single-view 3D Openable Part Detectioncited by 52

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 20; 2026: 9.

33 papers.

Sort:

By area:

  1. 2023An Outlook into the Future of Egocentric Visioncited by 84
  2. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  3. 2025ArticulatedGS: Self-supervised Digital Twin Modeling of Articulated Objects using 3D Gaussian Splattingcited by 35
  4. 2025EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligencecited by 32
  5. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  6. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  7. 2025DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Datacited by 21
  8. 2025Dexterous World Modelscited by 15
  9. 2024Survey on Modeling of Human-made Articulated Objectscited by 14
  10. 2025Beyond the Frame: Generating 360 Panoramic Videos from Perspective Videoscited by 14
  11. 2025GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulationcited by 14
  12. 2025HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Videocited by 13
21 more
  1. 2025RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Graspingcited by 11
  2. 20252HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videoscited by 9
  3. 2025Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priorscited by 9
  4. 2025TwinAligner: Visual-Dynamic Alignment Empowers Physics-aware Real2Sim2Real for Robotic Manipulationcited by 7
  5. 2025AffordanceSAM: Segment Anything Once More in Affordance Groundingcited by 6
  6. 2025Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scalecited by 6
  7. 2025Selective Contrastive Learning for Weakly Supervised Affordance Groundingcited by 6
  8. 2025REACT3D: Recovering Articulations for Interactive Physical 3D Scenescited by 6
  9. 2026AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignmentcited by 5
  10. 2026Whareformer: Learning to Track What is Where in Long Egocentric Videoscited by 2
  11. 2024Opening Articulated Structures in the Real Worldcited by 1
  12. 2025Visual Affordance Prediction: Survey and Reproducibilitycited by 1
  13. 2025Part2GS: Part-aware Modeling of Articulated Objects using 3D Gaussian Splattingcited by 1
  14. 2025GAOT: Generating Articulated Objects Through Text-Guided Diffusion Modelscited by 1
  15. 2026AffordanceGrasp-R1:Leveraging Reasoning-Based Affordance Segmentation with Reinforcement Learning for Robotic Graspingcited by 1
  16. 2026Panoramic Affordance Predictioncited by 1
  17. 2026PhysGraph: A Physics-aware 3D Scene Graph for Perception and Reasoningcited by 1
  18. 2026GSAM: A Generalizable and Safe Robotic Framework for Articulated Object Manipulationcited by 0
  19. 2026GROW2: Grounding Which and Where for Robot Tool Usecited by 0
  20. 2026VLAff: Vision-Language-Affordance Model for Unified Actionable Affordancescited by 0
  21. 2026Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arenacited by 0