# PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph

arXiv: https://arxiv.org/abs/2608.01905 · HTML: https://mlsanity.com/paper/2608.01905/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 47 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Builds on (most cited first, 30 of 36)

- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation](https://mlsanity.com/paper/2210.02697/) (2022)
- [UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy](https://mlsanity.com/paper/2303.00938/) (2023)
- [Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos](https://mlsanity.com/paper/2507.15597/) (2025)
- [Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction](https://mlsanity.com/paper/2404.00562/) (2024)
- [Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos](https://mlsanity.com/paper/2510.21571/) (2025)
- [DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions](https://mlsanity.com/paper/2403.17827/) (2024)
- [GraspXL: Generating Grasping Motions for Diverse Objects at Scale](https://mlsanity.com/paper/2403.19649/) (2024)
- [ContactGen: Generative Contact Modeling for Grasp Generation](https://mlsanity.com/paper/2310.03740/) (2023)
- [Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands](https://mlsanity.com/paper/2208.12250/) (2022)
- [HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Models](https://mlsanity.com/paper/2503.19157/) (2025)
- [InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2509.09555/) (2025)
- [MotionChain: Conversational Motion Controllers via Multimodal Prompts](https://mlsanity.com/paper/2404.01700/) (2024)
- [BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects](https://mlsanity.com/paper/2412.05066/) (2024)
- [SemGrasp: Semantic Grasp Generation via Language Aligned Discretization](https://mlsanity.com/paper/2404.03590/) (2024)
- [DexGrasp Anything: Towards Universal Robotic Dexterous Grasping with Physics Awareness](https://mlsanity.com/paper/2503.08257/) (2025)
- [ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusion](https://mlsanity.com/paper/2409.09300/) (2024)
- [AnyDexGrasp: General Dexterous Grasping for Different Hands with Human-level Learning Efficiency](https://mlsanity.com/paper/2502.16420/) (2025)
- [Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamics](https://mlsanity.com/paper/2409.04033/) (2024)
- [EgoSim: Egocentric World Simulator for Embodied Interaction Generation](https://mlsanity.com/paper/2604.01001/) (2026)
- [Human-Object Interaction via Automatically Designed VLM-Guided Motion Policy](https://mlsanity.com/paper/2503.18349/) (2025)
- [SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios](https://mlsanity.com/paper/2506.02444/) (2025)
- [GazeVLA: Learning Human Intention for Robotic Manipulation](https://mlsanity.com/paper/2604.22615/) (2026)
- [Dynamic Reconstruction of Hand-Object Interaction with Distributed Force-aware Contact Representation](https://mlsanity.com/paper/2411.09572/) (2024)
- [TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactions](https://mlsanity.com/paper/2510.14874/) (2025)

## Abstract (arXiv)

Hand-object interaction (HOI) is a fundamental human behavior with broad applications in AR/VR, digital humans, and embodied interaction. Existing methods typically require predefined object geometry, object trajectories, or task-specific conditions, limiting their use with natural real-world inputs. To address this, we study a more practical problem of synthesizing 3D hand-object interaction sequences from a single RGB photograph and an open-vocabulary language instruction, and introduce PhotoHOI. PhotoHOI first uses a vision-language model to parse the input image and instruction into a structured task specification, including the interaction object, target region, and spatial relation. It then recovers a compact task-relevant 3D scene and plans a smooth collision-aware object trajectory based on the recovered object states, support relations, and surrounding scene geometry. To synthesize hand motion that generalizes to real-world photographs and unseen objects, it learns transferable task-conditioned contact and contact-conditioned grasp priors from large-scale affordance and HOI data. The grasp is further refined in a learned latent space, constraining the optimization to a plausible hand-pose manifold. Experiments on GRAB and H2O demonstrate improved contact quality and reduced penetration over representative baselines. Results on real-world photographs further demonstrate higher task success and scene consistency, together with generalization to unseen objects and open-vocabulary instructions.