mlsanityData to 7 Oct 2026

GenHeld: Generating and Editing Handheld Objects

arXiv 2406.05059 ↗First posted 7 Jun 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Grasping is an important human activity that has long been studied in robotics, computer vision, and cognitive science. Most existing works study grasping from the perspective of synthesizing hand poses conditioned on 3D or 2D object representations. We propose GenHeld to address the inverse problem of synthesizing held objects conditioned on 3D hand model or 2D image. Given a 3D model of hand, GenHeld 3D can select a plausible held object from a large dataset using compact object representations called object codes.The selected object is then positioned and oriented to form a plausible grasp without changing hand pose. If only a 2D hand image is available, GenHeld 2D can edit this image to add or replace a held object. GenHeld 2D operates by combining the abilities of GenHeld 3D with diffusion-based image editing. Results and experiments show that we outperform baselines and can generate plausible held objects in both 2D and 3D. Our experiments demonstrate that our method achieves high quality and plausibility of held object synthesis in both 3D and 2D.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 82 of its 108 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

82 of 108 references.

References · 82

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2023Segment Anythingcited by 8,158
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
38 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  4. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  5. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  6. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  7. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  8. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  9. 2022Activating More Pixels in Image Super-Resolution Transformercited by 433
  10. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  11. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  12. 2022Kubric: A scalable dataset generatorcited by 373
  13. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  14. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  15. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  16. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 254
  17. 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 246
  18. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 193
  19. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  20. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  21. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  22. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  23. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  24. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  25. 2022DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulationcited by 129
  26. 2020CPF: Learning a Contact Potential Field to Model the Hand-Object Interactioncited by 127
  27. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusioncited by 125
  28. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 122
  29. 2020We are More than Our Joints: Predicting how 3D Bodies Movecited by 115
  30. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  31. 2021D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactionscited by 109
  32. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 101
  33. 2021SAGA: Stochastic Whole-Body Grasping with Contactcited by 89
  34. 2024DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editingcited by 88
  35. 2023Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Renderingcited by 84
  36. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  37. 2022Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applicationscited by 78
  38. 2018GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learningcited by 77

50 most cited of 82 All 82 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Joint Diffusion for Universal Hand-Object Grasp Generation
  2. 2024FastGrasp: Efficient Grasp Synthesis with Diffusion
  3. 2025Contact Map Transfer with Conditional Diffusion Model for Generalizable Dexterous Grasp Generation
  4. 2026DCGrasp: Distance-aware Controllable Grasp Generation
  5. 2024Constrained 6-DoF Grasp Generation on Complex Shapes for Improved Dual-Arm Manipulation
  6. 2026Human Universal Grasping
  7. 2024DexGraspNet 2.0: Learning Generative Dexterous Grasping in Large-scale Synthetic Cluttered Scenes
  8. 2025G-DexGrasp: Generalizable Dexterous Grasping Synthesis Via Part-Aware Prior Retrieval and Prior-Assisted Generation