mlsanityData to 7 Oct 2026

Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors

arXiv 2209.02485 ↗First posted 6 Sep 2022cs.CV cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

We present a method for inferring diverse 3D models of human-object interactions from images. Reasoning about how humans interact with objects in complex scenes from a single 2D image is a challenging task given ambiguities arising from the loss of information through projection. In addition, modeling 3D interactions requires the generalization ability towards diverse object categories and interaction types. We propose an action-conditioned modeling of interactions that allows us to infer diverse 3D arrangements of humans and objects without supervision on contact regions or 3D scene geometry. Our method extracts high-level commonsense knowledge from large language models (such as GPT-3), and applies them to perform 3D reasoning of human-object interactions. Our key insight is priors extracted from large language models can help in reasoning about human-object contacts from textural prompts only. We quantitatively evaluate the inferred 3D models on a large human-object interaction dataset and show how our method leads to better 3D reconstructions. We further qualitatively evaluate the effectiveness of our method on real images and demonstrate its generalizability towards interaction types and object categories.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 6 Sep 2022.

Builds on

30 of 43 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  5. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  7. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  8. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  9. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  10. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  11. 2019PointRend: Image Segmentation as Renderingcited by 439
  12. 2017Neural 3D Mesh Renderercited by 437
18 more
  1. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  2. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  3. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  4. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  5. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  6. 2021SPEC: Seeing People in the Wild with an Estimated Cameracited by 134
  7. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  8. 2021CLIP-It! Language-Guided Video Summarizationcited by 99
  9. 2021Skill Induction and Planning with Latent Languagecited by 99
  10. 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
  11. 2022Human-Aware Object Placement for Visual Environment Reconstructioncited by 56
  12. 2020Holistic 3D Human and Scene Mesh Estimation from Single View Imagescited by 54
  13. 2018Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layerscited by 33
  14. 2020From Points to Multi-Object 3D Reconstructioncited by 29
  15. 2020Compositional Networks Enable Systematic Generalization for Grounded Language Understandingcited by 18
  16. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformerscited by 17
  17. 2017Natural Language Guided Visual Relationship Detectioncited by 15
  18. 2019Language-guided Semantic Mapping and Mobile Manipulation in Partially Observable Environmentscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 10; 2025: 7; 2026: 2.

30 papers.

Sort:

By area:

  1. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  2. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  3. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  4. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  5. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  6. 2023SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generationcited by 62
  7. 2023Visibility Aware Human-Object Interaction Tracking from Single RGB Cameracited by 58
  8. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 54
  9. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  10. 2023I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactionscited by 42
  11. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generationcited by 37
  12. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
18 more
  1. 2024F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactionscited by 24
  2. 2023Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from Monocular RGBD Streamcited by 23
  3. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 22
  4. 2025PICO: Reconstructing 3D People In Contact with Objectscited by 22
  5. 2024TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactionscited by 18
  6. 2025Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactionscited by 13
  7. 2025InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsingcited by 13
  8. 2023Compositional 3D Human-Object Neural Animationcited by 9
  9. 2024Monocular Human-Object Reconstruction in the Wildcited by 9
  10. 2025VolumetricSMPL: A Neural Volumetric Body Model for Efficient Interactions, Contacts, and Collisionscited by 9
  11. 2023Ins-HOI: Instance Aware Human-Object Interactions Recoverycited by 7
  12. 2023Primitive-based 3D Human-Object Interaction Modelling and Programmingcited by 7
  13. 2024Pose Priors from Language Modelscited by 6
  14. 2026Unleashing Guidance Without Classifiers for Human-Object Interaction Animationcited by 5
  15. 2024TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scenecited by 2
  16. 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 2
  17. 2025Realistic Clothed Human and Object Joint Reconstruction from a Single Imagecited by 0
  18. 2026Efficient Human-Contact Representation for Human-Scene Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025InteractVLM: 3D Interaction Reasoning from 2D Foundational Models
  2. 2025BOP-ASK: Object-Interaction Reasoning for Vision-Language Models
  3. 2026ViHOI: Human-Object Interaction Synthesis with Visual Priors
  4. 2023GenZI: Zero-Shot 3D Human-Scene Interaction Generation
  5. 2024Ask, Pose, Unite: Scaling Data Acquisition for Close Interactions with Vision Language Models
  6. 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
  7. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understanding
  8. 2026Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models