# Decomposed Vector-Quantized Variational Autoencoder for Human Grasp Generation

arXiv: https://arxiv.org/abs/2407.14062 · HTML: https://mlsanity.com/paper/2407.14062/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 19 Jul 2024.

## Builds on (most cited first, 27 of 27)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation](https://mlsanity.com/paper/2203.09516/) (2022)
- [3D Hand Shape and Pose from Images in the Wild](https://mlsanity.com/paper/1902.03451/) (2019)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [Grasping Field: Learning Implicit Representations for Human Grasps](https://mlsanity.com/paper/2008.04451/) (2020)
- [Hand-Object Contact Consistency Reasoning for Human Grasps Generation](https://mlsanity.com/paper/2104.03304/) (2021)
- [ContactOpt: Optimizing Contact to Improve Grasps](https://mlsanity.com/paper/2104.07267/) (2021)
- [Capturing Hands in Action using Discriminative Salient Points and Physics Simulation](https://mlsanity.com/paper/1506.02178/) (2015)
- [Attentive Relational Networks for Mapping Images to Scene Graphs](https://mlsanity.com/paper/1811.10696/) (2018)
- [ContactGen: Generative Contact Modeling for Grasp Generation](https://mlsanity.com/paper/2310.03740/) (2023)
- [A Skeleton-Driven Neural Occupancy Representation for Articulated Hands](https://mlsanity.com/paper/2109.11399/) (2021)
- [Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models](https://mlsanity.com/paper/2310.02242/) (2023)
- [3D Object Reconstruction from Hand-Object Interactions](https://mlsanity.com/paper/1704.00529/) (2017)
- [Symbiotic Attention with Privileged Information for Egocentric Action Recognition](https://mlsanity.com/paper/2002.03137/) (2020)
- [SGFormer: Semantic Graph Transformer for Point Cloud-based 3D Scene Graph Generation](https://mlsanity.com/paper/2303.11048/) (2023)
- [RGB-Depth Fusion GAN for Indoor Depth Completion](https://mlsanity.com/paper/2203.10856/) (2022)
- [Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning](https://mlsanity.com/paper/2310.19559/) (2023)
- [Adaptive Weight Learning for Multiple Outcome Optimization With Continuous Treatment](https://mlsanity.com/paper/2402.11092/) (2024)

## Built on by (most cited first, 4 of 4)

- [Physically Interpretable World Models via Weakly Supervised Representation Learning](https://mlsanity.com/paper/2412.12870/) (2024)
- [Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation](https://mlsanity.com/paper/2601.02778/) (2026)
- [AffordGrasp: Cross-Modal Diffusion for Affordance-Aware Grasp Synthesis](https://mlsanity.com/paper/2603.08021/) (2026)
- [Leveraging Extrinsic Dexterity for Occluded Grasping on Grasp Constraining Walls](https://mlsanity.com/paper/2507.14721/) (2025)

## Abstract (arXiv)

Generating realistic human grasps is a crucial yet challenging task for applications involving object manipulation in computer graphics and robotics. Existing methods often struggle with generating fine-grained realistic human grasps that ensure all fingers effectively interact with objects, as they focus on encoding hand with the whole representation and then estimating both hand posture and position in a single step. In this paper, we propose a novel Decomposed Vector-Quantized Variational Autoencoder (DVQ-VAE) to address this limitation by decomposing hand into several distinct parts and encoding them separately. This part-aware decomposed architecture facilitates more precise management of the interaction between each component of hand and object, enhancing the overall reality of generated human grasps. Furthermore, we design a newly dual-stage decoding strategy, by first determining the type of grasping under skeletal physical constraints, and then identifying the location of the grasp, which can greatly improve the verisimilitude as well as adaptability of the model to unseen hand-object interaction. In experiments, our model achieved about 14.1% relative improvement in the quality index compared to the state-of-the-art methods in four widely-adopted benchmarks. Our source code is available at https://github.com/florasion/D-VQVAE.