mlsanityData to 14 Oct 2026

LEMON: Learning 3D Human-Object Interaction Relation from 2D Images

arXiv 2312.08963 ↗First posted 14 Dec 2023cs.CVcited by 49

Not on the map. Why?

Abstract (arXiv)

Learning 3D human-object interaction relation is pivotal to embodied AI and interaction modeling. Most existing methods approach the goal by learning to predict isolated interaction elements, e.g., human contact, object affordance, and human-object spatial relation, primarily from the perspective of either the human or the object. Which underexploit certain correlations between the interaction counterparts (human and object), and struggle to address the uncertainty in interactions. Actually, objects' functionalities potentially affect humans' interaction intentions, which reveals what the interaction is. Meanwhile, the interacting humans and objects exhibit matching geometric structures, which presents how to interact. In light of this, we propose harnessing these inherent correlations between interaction counterparts to mitigate the uncertainty and jointly anticipate the above interaction elements in 3D space. To achieve this, we present LEMON (LEarning 3D huMan-Object iNteraction relation), a unified model that mines interaction intentions of the counterparts and employs curvatures to guide the extraction of geometric correlations, combining them to anticipate the interaction elements. Besides, the 3D Interaction Relation dataset (3DIR) is collected to serve as the test bed for training and evaluation. Extensive experiments demonstrate the superiority of LEMON over methods estimating each element in isolation.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 79 of its 128 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

79 of 128 references.

References · 79

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023Segment Anythingcited by 8,234
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  6. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  7. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  8. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  10. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  11. 2018Datasheets for Datasetscited by 1,386
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
38 more
  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  2. 2017End-to-end Recovery of Human Shape and Posecited by 885
  3. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 526
  4. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  5. 2021Conditional DETR for Fast Training Convergencecited by 424
  6. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 311
  7. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 295
  8. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  9. 2018Generating 3D faces using Convolutional Mesh Autoencoderscited by 268
  10. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
  11. 2017Learning to Detect Human-Object Interactionscited by 240
  12. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  13. 2023Object Motion Guided Human Motion Synthesiscited by 212
  14. 2021AGORA: Avatars in Geography Optimized for Regression Analysiscited by 205
  15. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  16. 2015Visual Semantic Role Labelingcited by 190
  17. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  18. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 182
  19. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
  20. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 180
  21. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  22. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  23. 2022GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Partscited by 139
  24. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
  25. 2023One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformercited by 130
  26. 2018Grounded Human-Object Interaction Hotspots from Videocited by 129
  27. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 128
  28. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 128
  29. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  30. 20213D AffordanceNet: A Benchmark for Visual Object Affordance Understandingcited by 100
  31. 2020Contact and Human Dynamics from Monocular Videocited by 95
  32. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  33. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 94
  34. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  35. 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 93
  36. 2020Learning Dexterous Grasping with Object-Centric Visual Affordancescited by 85
  37. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 84
  38. 2022AKB-48: A Real-World Articulated Object Knowledge Basecited by 82

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 22; 2026: 14.

49 papers.

Sort:

By area:

  1. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 72
  2. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
  3. 2024DisPose: Disentangling Pose Guidance for Controllable Human Image Animationcited by 35
  4. 2024Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Modelscited by 32
  5. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 32
  6. 2024GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Groundingcited by 28
  7. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  8. 2025PICO: Reconstructing 3D People In Contact with Objectscited by 22
  9. 2025InteractVLM: 3D Interaction Reasoning from 2D Foundational Modelscited by 21
  10. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 20
  11. 2025HERO: Human Reaction Generation from Videoscited by 15
  12. 2025Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactionscited by 13
37 more
  1. 2025SIGMAN:Scaling 3D Human Gaussian Generation with Millions of Assetscited by 12
  2. 2024Grounding 3D Scene Affordance From Egocentric Interactionscited by 11
  3. 2024PEAR: Phrase-Based Hand-Object Interaction Anticipationcited by 10
  4. 2024FIction: 4D Future Interaction Prediction from Videocited by 9
  5. 2025Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priorscited by 9
  6. 2025TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactionscited by 8
  7. 2024FORCE: Physics-aware Human-object Interactioncited by 7
  8. 2025HOI-PAGE: Zero-Shot Human-Object Interaction Generation with Part Affordance Guidancecited by 6
  9. 2025DAViD: Modeling Dynamic Affordance of 3D Objects Using Pre-trained Video Diffusion Modelscited by 4
  10. 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 3
  11. 2025GRACE: Estimating Geometry-level 3D Human-Scene Contact from 2D Imagescited by 3
  12. 2025Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videoscited by 3
  13. 2025O3Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulationcited by 3
  14. 2025Efficient and Scalable Monocular Human-Object Interaction Motion Reconstructioncited by 3
  15. 2025Decoupled Generative Modeling for Human-Object Interaction Synthesiscited by 3
  16. 2026Gloria: Consistent Character Video Generation via Content Anchorscited by 3
  17. 2026VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Modelcited by 2
  18. 2026Towards in-the-wild Egocentric 3D Hand-Object Pose Estimationcited by 2
  19. 2026Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generationcited by 2
  20. 2024Affordance Labeling and Exploration: A Manifold-Based Approachcited by 1
  21. 2024Ask, Pose, Unite: Scaling Data Acquisition for Close Interactions with Vision Language Modelscited by 1
  22. 2025Reference-free Human-Object Interaction Editingcited by 1
  23. 2025Contact-Aware Amodal Completion for Human-Object Interaction via Multi-Regional Inpaintingcited by 1
  24. 2025Object Affordance Recognition and Grounding via Multi-scale Cross-modal Representation Learningcited by 1
  25. 2025Monocular Visual 8D Pose Estimation for Articulated Bicycles and Cyclistscited by 1
  26. 2025DecoDINO: 3D Human-Scene Contact Prediction with Semantic Classificationcited by 1
  27. 2026TeHOR: Text-Guided 3D Human and Object Reconstruction with Texturescited by 1
  28. 2026HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Groundingcited by 1
  29. 2026End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstructioncited by 1
  30. 2026GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systemscited by 1
  31. 2026LogiShot: Logically Coherent Cross-Shot Video Generationcited by 1
  32. 2025VPHO: Joint Visual-Physical Cue Learning and Aggregation for Hand-Object Pose Estimationcited by 0
  33. 2026Pi-HOC: Pairwise 3D Human-Object Contact Estimationcited by 0
  34. 2026SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergycited by 0
  35. 2026LEXIS: LatEnt ProXimal Interaction Signatures for 3D HOI from an Imagecited by 0
  36. 2026Reconstructing Humans and Objects in Interaction using Large Reconstruction Modelscited by 0
  37. 2026GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Imagecited by 0