mlsanityData to 7 Oct 2026

Ask, Pose, Unite: Scaling Data Acquisition for Close Interactions with Vision Language Models

arXiv 2410.00309 ↗First posted 1 Oct 2024cs.CV cs.AI cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Social dynamics in close human interactions pose significant challenges for Human Mesh Estimation (HME), particularly due to the complexity of physical contacts and the scarcity of training data. Addressing these challenges, we introduce a novel data generation method that utilizes Large Vision Language Models (LVLMs) to annotate contact maps which guide test-time optimization to produce paired image and pseudo-ground truth meshes. This methodology not only alleviates the annotation burden but also enables the assembly of a comprehensive dataset specifically tailored for close interactions in HME. Our Ask Pose Unite (APU) dataset, comprising over 6.2k human mesh pairs in contact covering diverse interaction types, is curated from images depicting naturalistic person-to-person scenes. We empirically show that using our dataset to train a diffusion-based contact prior, used as guidance during optimization, improves mesh estimation on unseen interactions. Our work addresses longstanding challenges of data scarcity for close interactions in HME enhancing the field's capabilities of handling complex interaction scenarios.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 68 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

59 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ChatPose: Chatting about 3D Human Pose30 Nov

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  5. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  6. 2017End-to-end Recovery of Human Shape and Posecited by 883
  7. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  8. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  9. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  10. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  11. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  12. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
38 more
  1. 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
  2. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  3. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  4. 2022CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimationcited by 237
  5. 2023BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motioncited by 229
  6. 2020Monocular, One-stage, Regression of Multiple 3D Peoplecited by 216
  7. 2021AGORA: Avatars in Geography Optimized for Regression Analysiscited by 203
  8. 2023WHAM: Reconstructing World-grounded Humans with Accurate 3D Motioncited by 200
  9. 2017Single-Shot Multi-Person 3D Pose Estimation From Monocular RGBcited by 195
  10. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  11. 2020Populating 3D Scenes by Learning Human-Scene Interactioncited by 164
  12. 2023SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimationcited by 154
  13. 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
  14. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  15. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  16. 2021Putting People in their Place: Monocular Regression of 3D People in Depthcited by 141
  17. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  18. 2020Coherent Reconstruction of Multiple Humans from a Single Imagecited by 136
  19. 2024Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shotcited by 98
  20. 2023TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environmentscited by 89
  21. 2021On Self-Contact and Human Posecited by 88
  22. 2023Hi4D: 4D Instance Segmentation of Close Human Interactioncited by 87
  23. 2021Multi-Person Extreme Motion Predictioncited by 74
  24. 2021DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Camerascited by 67
  25. 2021Learning to Estimate Robust 3D Human Mesh from In-the-Wild Crowded Scenescited by 64
  26. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  27. 2023DECO: Dense Estimation of 3D Human-Scene Contact In The Wildcited by 54
  28. 2022Occluded Human Mesh Recoverycited by 52
  29. 2023LEMON: Learning 3D Human-Object Interaction Relation from 2D Imagescited by 49
  30. 2023ReFit: Recurrent Fitting Network for 3D Human Recoverycited by 47
  31. 2023Generative Proxemics: A Prior for 3D Social Interaction from Imagescited by 43
  32. 2023Detecting Human-Object Contact in Imagescited by 41
  33. 2024AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimationcited by 41
  34. 2024LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesiscited by 39
  35. 2022Embodied Scene-aware Human Pose Estimationcited by 28
  36. 2023CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Imagescited by 19
  37. 2023DPoser: Diffusion Model as Robust 3D Human Pose Priorcited by 17
  38. 2024DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recoverycited by 16

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Anny-Fit: All-Age Human Mesh Recoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025InteractVLM: 3D Interaction Reasoning from 2D Foundational Models
  2. 2025InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing
  3. 2026Pi-HOC: Pairwise 3D Human-Object Contact Estimation
  4. 2024DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors
  5. 2023GenZI: Zero-Shot 3D Human-Scene Interaction Generation
  6. 2022Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors
  7. 2026COSMI: COmpositional Synthesis of Multi-object Interactions
  8. 2023Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation