mlsanityData to 7 Oct 2026

JointNet: Extending Text-to-Image Diffusion for Dense Distribution Modeling

arXiv 2310.06347 ↗First posted 10 Oct 2023cs.CVcited by 23

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We introduce JointNet, a novel neural network architecture for modeling the joint distribution of images and an additional dense modality (e.g., depth maps). JointNet is extended from a pre-trained text-to-image diffusion model, where a copy of the original network is created for the new dense modality branch and is densely connected with the RGB branch. The RGB branch is locked during network fine-tuning, which enables efficient learning of the new modality distribution while maintaining the strong generalization ability of the large-scale pre-trained diffusion model. We demonstrate the effectiveness of JointNet by using RGBD diffusion as an example and through extensive experiments, showcasing its applicability in a variety of applications, including joint RGBD generation, dense depth prediction, depth-conditioned image generation, and coherent tile-based 3D panorama generation.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 29 of its 34 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

29 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions8 Jun
  2. 2023LDM3D: Latent Diffusion Model for 3D18 May
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  4. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2013Auto-Encoding Variational Bayescited by 3,980
13 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  2. 2016Improved Techniques for Training GANscited by 2,938
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  5. 2021Vision Transformers for Dense Predictioncited by 1,789
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2016HyperNetworkscited by 1,003
  8. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  9. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  10. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  11. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  12. 20223D Common Corruptions and Data Augmentationcited by 120
  13. 2023Mixture of Diffusers for scene composition and high resolution image generationcited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 9; 2026: 8.

23 papers.

Sort:

By area:

  1. 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 283
  2. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  3. 2023Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusioncited by 50
  4. 2025Matrix3D: Large Photogrammetry Model All-in-Onecited by 39
  5. 2024A Simple Approach to Unifying Diffusion-based Conditional Generationcited by 13
  6. 2025JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformerscited by 11
  7. 2025Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generationcited by 8
  8. 2025FICGen: Frequency-Inspired Contextual Disentanglement for Layout-driven Degraded Image Generationcited by 4
  9. 2024Comprehensive Generative Replay for Task-Incremental Segmentation with Concurrent Appearance and Semantic Forgettingcited by 3
  10. 2026UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priorscited by 3
  11. 2026Modality Forcing for Scalable Spatial Generationcited by 3
  12. 2025More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Modelscited by 2
11 more
  1. 2024Diffusion-based Data Augmentation and Knowledge Distillation with Generated Soft Labels Solving Data Scarcity Problems of SAR Oil Spill Segmentationcited by 1
  2. 2025Joint Learning of Depth and Appearance for Portrait Image Animationcited by 1
  3. 2025Perfecting Depth: Uncertainty-Aware Enhancement of Metric Depthcited by 1
  4. 2026Composing People Together: Iterative Pose-Image Generation for Multi-Person Interaction Scenescited by 1
  5. 2025Enhancing Event-based Object Detection with Monocular Normal Mapscited by 0
  6. 2025LumiX: Structured and Coherent Text-to-Intrinsic Generationcited by 0
  7. 2026GeoDiff4D: Geometry-Aware Diffusion for 4D Head Avatar Reconstructioncited by 0
  8. 2026UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perceptioncited by 0
  9. 2026Enhancing In-context Panoramic Generation via Geometric-aware Pretrainingcited by 0
  10. 2026SketchSense: Learning to Interpret Imperfect Sketch Guidance for Image Inpaintingcited by 0
  11. 2026SCULPT: Subtractive Composition for 3D Part Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Merging and Splitting Diffusion Paths for Semantically Coherent Panoramas
  2. 2023Efficient Multimodal Diffusion Models Using Joint Data Infilling with Partially Shared U-Net
  3. 2024Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions
  4. 2023PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with Time-Decoupled Training and Reusable Coop-Diffusion
  5. 2025Orchid: Image Latent Diffusion for Joint Appearance and Geometry Generation
  6. 2024Panoptic Diffusion Models: co-generation of images and segmentation maps
  7. 2024Zippo: Zipping Color and Transparency Distributions into a Single Diffusion Model
  8. 2023Exploiting Diffusion Prior for Generalizable Dense Prediction