# 4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction

arXiv: https://arxiv.org/abs/2610.08782 · HTML: https://mlsanity.com/paper/2610.08782/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 21 of its 72 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 21 of 21)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation](https://mlsanity.com/paper/2501.12202/) (2025)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [Let us Build Bridges: Understanding and Extending Diffusion Generative Models](https://mlsanity.com/paper/2208.14699/) (2022)
- [GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion](https://mlsanity.com/paper/2402.14810/) (2024)
- [HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction](https://mlsanity.com/paper/2406.06843/) (2024)
- [Do as I Do: Dexterous Manipulation Data from Everyday Human Videos](https://mlsanity.com/paper/2606.19333/) (2026)
- [WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos](https://mlsanity.com/paper/2602.22209/) (2026)
- [ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videos](https://mlsanity.com/paper/2602.06226/) (2026)
- [ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions](https://mlsanity.com/paper/2603.25791/) (2026)
- [Reconstructing Objects along Hand Interaction Timelines in Egocentric Video](https://mlsanity.com/paper/2512.07394/) (2025)
- [GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting](https://mlsanity.com/paper/2603.18912/) (2026)
- [Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions](https://mlsanity.com/paper/2604.12929/) (2026)
- [Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction](https://mlsanity.com/paper/2605.20992/) (2026)
- [Zero-shot Reconstruction of In-Scene Object Manipulation from Video](https://mlsanity.com/paper/2512.19684/) (2025)
- [AffordGrasp: Cross-Modal Diffusion for Affordance-Aware Grasp Synthesis](https://mlsanity.com/paper/2603.08021/) (2026)
- [AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos](https://mlsanity.com/paper/2609.24487/) (2026)

## Abstract (arXiv)

Existing methods for 4D hand-object reconstruction often rely on costly per-sequence optimization, while generative approaches typically synthesize interactions from random noise, which can lead to unstable interaction prediction. We introduce 4D-HOF, a feed-forward framework that reconstructs 4D hand-object interactions from coarse but informative estimates produced by vision foundation models. Concretely, we learn a conditional flow matching model that transports foundation-model-derived hand-object states toward an interaction manifold, allowing the model to correct errors in translation, rotation, and alignment in a feed-forward manner. A key advantage of our generative formulation is that it naturally enables test-time guidance within the transport process. Rather than applying a separate post-hoc optimization after reconstruction, we directly steer the evolving generative states using physical interaction constraints and observed 2D evidence, allowing the reconstruction to be refined as part of the generative process itself. By training the generative model on diverse datasets, 4D-HOF generalizes robustly to challenging in-the-wild scenarios. Experiments on out-of-domain benchmarks show that 4D-HOF achieves state-of-the-art performance, producing more stable and accurate 4D hand-object reconstructions.