mlsanityData to 7 Oct 2026

DiffSurf: A Transformer-based Diffusion Model for Generating and Reconstructing 3D Surfaces in Pose

arXiv 2408.14860 ↗First posted 27 Aug 2024cs.CVcited by 2

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

This paper presents DiffSurf, a transformer-based denoising diffusion model for generating and reconstructing 3D surfaces. Specifically, we design a diffusion transformer architecture that predicts noise from noisy 3D surface vertices and normals. With this architecture, DiffSurf is able to generate 3D surfaces in various poses and shapes, such as human bodies, hands, animals and man-made objects. Further, DiffSurf is versatile in that it can address various 3D downstream tasks including morphing, body shape variation and 3D human mesh fitting to 2D keypoints. Experimental results on 3D human model benchmarks demonstrate that DiffSurf can generate shapes with greater diversity and higher quality than previous generative models. Furthermore, when applied to the task of single-image 3D human mesh recovery, DiffSurf achieves accuracy comparable to prior techniques at a near real-time rate.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 71 of its 93 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

71 of 93 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023PolyDiff: Generating 3D Polygonal Meshes with Diffusion Models18 Dec
  2. 2023Generative Approach for Probabilistic Human Mesh Recovery using Diffusion Models5 Aug
  3. 2023DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation4 Jul
  4. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  5. 2023Controllable Mesh Generation Through Sparse Latent Point Diffusion Models14 Mar
  6. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  8. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  9. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  10. 2021A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion7 Dec
  11. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 60

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  9. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  10. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  11. 2017End-to-end Recovery of Human Shape and Posecited by 883
  12. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
38 more
  1. 2019Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loopcited by 625
  2. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  3. 2016Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervisioncited by 493
  4. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  5. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  6. 2020End-to-End Human Pose and Mesh Reconstruction with Transformerscited by 417
  7. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  8. 2017Learning from Synthetic Humanscited by 358
  9. 2021PARE: Part Attention Regressor for 3D Human Body Estimationcited by 334
  10. 20163D Menagerie: Modeling the 3D shape and pose of animalscited by 310
  11. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  12. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  13. 2019Convolutional Mesh Regression for Single-Image Human Shape Reconstructioncited by 270
  14. 2018Generating 3D faces using Convolutional Mesh Autoencoderscited by 267
  15. 2021Mesh Graphormercited by 264
  16. 2021PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loopcited by 250
  17. 2020Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Posecited by 248
  18. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  19. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  20. 2021Shape As Points: A Differentiable Poisson Solvercited by 186
  21. 20183D-CODED : 3D Correspondences by Deep Deformationcited by 160
  22. 2021Probabilistic Modeling for Human Mesh Recoverycited by 152
  23. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  24. 2022Recovering 3D Human Mesh from Monocular Images: A Surveycited by 103
  25. 2022DiffPose: Toward More Reliable 3D Pose Estimationcited by 99
  26. 2020Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flowscited by 94
  27. 2022Pose-NDF: Modeling Human Pose Manifolds with Neural Distance Fieldscited by 85
  28. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  29. 2020Human Body Model Fitting by Learned Gradient Descentcited by 81
  30. 2022Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformerscited by 81
  31. 2023Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregationcited by 81
  32. 2020Hierarchical Kinematic Human Mesh Recoverycited by 71
  33. 2017Variational Autoencoders for Deforming 3D Mesh Modelscited by 69
  34. 2020Unsupervised Shape and Pose Disentanglement for 3D Meshescited by 61
  35. 2020LIMP: Learning Latent Shape Representations with Metric Preservation Priorscited by 54
  36. 2022Occluded Human Mesh Recoverycited by 52
  37. 2022BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Informationcited by 51
  38. 2023Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shapecited by 43

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025MeshMamba: State Space Models for Articulated 3D Mesh Generation and Reconstructioncited by 1
  2. 2026Ordered Diffusion for 3D Human Registrationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025LieHMR: Autoregressive Human Mesh Recovery with SO(3) Diffusion
  2. 2024DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans
  3. 2022DiffPose: Multi-hypothesis Human Pose Estimation using Diffusion models
  4. 2023DiffHPE: Robust, Coherent 3D Human Pose Lifting with Diffusion
  5. 2024DiffTF++: 3D-aware Diffusion Transformer for Large-Vocabulary 3D Generation
  6. 2024D3PRefiner: A Diffusion-based Denoise Method for 3D Human Pose Refinement
  7. 2023DPoser: Diffusion Model as Robust 3D Human Pose Prior
  8. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation