# A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervision

arXiv: https://arxiv.org/abs/2412.00623 · HTML: https://mlsanity.com/paper/2412.00623/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 66 of its 74 references that are in the map.
- First posted on arXiv 1 Dec 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 66)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [Universal Guidance for Diffusion Models](https://mlsanity.com/paper/2302.07121/) (2023)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors](https://mlsanity.com/paper/2306.17843/) (2023)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Splatter Image: Ultra-Fast Single-View 3D Reconstruction](https://mlsanity.com/paper/2312.13150/) (2023)
- [GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting](https://mlsanity.com/paper/2404.19702/) (2024)
- [ReconFusion: 3D Reconstruction with Diffusion Priors](https://mlsanity.com/paper/2312.02981/) (2023)
- [Novel View Synthesis with Diffusion Models](https://mlsanity.com/paper/2210.04628/) (2022)
- [3D Neural Field Generation using Triplane Diffusion](https://mlsanity.com/paper/2211.16677/) (2022)
- [Generative Novel View Synthesis with 3D-Aware Diffusion Models](https://mlsanity.com/paper/2304.02602/) (2023)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model](https://mlsanity.com/paper/2311.09217/) (2023)

## Built on by (most cited first, 9 of 9)

- [GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation](https://mlsanity.com/paper/2512.23180/) (2025)
- [FRIDU: Functional Map Refinement with Guided Image Diffusion](https://mlsanity.com/paper/2506.14322/) (2025)
- [Appreciate the View: A Task-Aware Evaluation Framework for Novel View Synthesis](https://mlsanity.com/paper/2511.12675/) (2025)
- [Complete Gaussian Splats from a Single Image with Denoising Diffusion Models](https://mlsanity.com/paper/2508.21542/) (2025)
- [Realiz3D: 3D Generation Made Photorealistic via Domain-Aware Learning](https://mlsanity.com/paper/2605.13852/) (2026)
- [GS-Voxel: Fitting-Free Structured Latents for Large-Scale 3DGS Generation](https://mlsanity.com/paper/2608.17988/) (2026)
- [Sparse auto-regressive modeling for scene generation from multi-view images](https://mlsanity.com/paper/2609.03931/) (2026)
- [VideoMDM: Towards 3D Human Motion Generation From 2D Supervision](https://mlsanity.com/paper/2606.13364/) (2026)
- [GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation](https://mlsanity.com/paper/2608.16234/) (2026)

## Abstract (arXiv)

We present a novel framework for training 3D image-conditioned diffusion models using only 2D supervision. Recovering 3D structure from 2D images is inherently ill-posed due to the ambiguity of possible reconstructions, making generative models a natural choice. However, most existing 3D generative models rely on full 3D supervision, which is impractical due to the scarcity of large-scale 3D datasets. To address this, we propose leveraging sparse-view supervision as a scalable alternative. While recent reconstruction models use sparse-view supervision with differentiable rendering to lift 2D images to 3D, they are predominantly deterministic, failing to capture the diverse set of plausible solutions and producing blurry predictions in uncertain regions. A key challenge in training 3D diffusion models with 2D supervision is that the standard training paradigm requires both the denoising process and supervision to be in the same modality. We address this by decoupling the noisy samples being denoised from the supervision signal, allowing the former to remain in 3D while the latter is provided in 2D. Our approach leverages suboptimal predictions from a deterministic image-to-3D model-acting as a "teacher"-to generate noisy 3D inputs, enabling effective 3D diffusion training without requiring full 3D ground truth. We validate our framework on both object-level and scene-level datasets, using two different 3D Gaussian Splat (3DGS) teachers. Our results show that our approach consistently improves upon these deterministic teachers, demonstrating its effectiveness in scalable and high-fidelity 3D generative modeling. See our project page at https://lesson-in-splats.github.io/