# PuzzleFusion: Unleashing the Power of Diffusion Models for Spatial Puzzle Solving

arXiv: https://arxiv.org/abs/2211.13785 · HTML: https://mlsanity.com/paper/2211.13785/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 36 of its 72 references that are in the map.
- First posted on arXiv 24 Nov 2022.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.14951/) (2021)
- [Diffusion Models for Implicit Image Segmentation Ensembles](https://mlsanity.com/paper/2112.03145/) (2021)
- [Learning to Find Good Correspondences](https://mlsanity.com/paper/1711.05971/) (2017)
- [EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations](https://mlsanity.com/paper/2207.06635/) (2022)
- [UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2104.05358/) (2021)
- [Star-Transformer](https://mlsanity.com/paper/1902.09113/) (2019)
- [Dual-Resolution Correspondence Networks](https://mlsanity.com/paper/2006.08844/) (2020)
- [HouseDiffusion: Vector Floorplan Generation via a Diffusion Model with Discrete and Continuous Denoising](https://mlsanity.com/paper/2211.13287/) (2022)

## Built on by (most cited first, 17 of 17)

- [PuzzleFusion++: Auto-agglomerative 3D Fracture Assembly by Denoise and Verify](https://mlsanity.com/paper/2406.00259/) (2024)
- [Rectified Point Flow: Generic Point Cloud Pose Estimation](https://mlsanity.com/paper/2506.05282/) (2025)
- [Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation](https://mlsanity.com/paper/2504.06961/) (2025)
- [Video Models Can Reason with Verifiable Rewards](https://mlsanity.com/paper/2605.15458/) (2026)
- [Differentiable and Learnable Wireless Simulation with Geometric Transformers](https://mlsanity.com/paper/2406.14995/) (2024)
- [Pairwise Alignment & Compatibility for Arbitrarily Irregular Image Fragments](https://mlsanity.com/paper/2507.09767/) (2025)
- [WAFFLE: Multimodal Floorplan Understanding in the Wild](https://mlsanity.com/paper/2412.00955/) (2024)
- [HouseTune: Two-Stage Floorplan Generation with LLM Assistance](https://mlsanity.com/paper/2411.12279/) (2024)
- [TPIE: Topology-Preserved Image Editing With Text Instructions](https://mlsanity.com/paper/2411.16714/) (2024)
- [Interact3D: Compositional 3D Generation of Interactive Objects](https://mlsanity.com/paper/2603.16085/) (2026)
- [GFPack++: Improving 2D Irregular Packing by Learning Gradient Field with Attention](https://mlsanity.com/paper/2406.07579/) (2024)
- [Nash Meets Wertheimer: Using Good Continuation in Jigsaw Puzzles](https://mlsanity.com/paper/2410.16857/) (2024)
- [BADGR: Bundle Adjustment Diffusion Conditioned by GRadients for Wide-Baseline Floor Plan Reconstruction](https://mlsanity.com/paper/2503.19340/) (2025)
- [ADiff4TPP: Asynchronous Diffusion Models for Temporal Point Processes](https://mlsanity.com/paper/2504.20411/) (2025)
- [ReassembleNet: Learnable Keypoints and Diffusion for 2D Fresco Reconstruction](https://mlsanity.com/paper/2505.21117/) (2025)
- [Unified Vector Floorplan Generation via Markup Representation](https://mlsanity.com/paper/2604.04859/) (2026)
- [Mitigating Domain Shift in Conditioned Floor Plan Generation: Synthetic Pre-training for Data-Efficient Adaptation](https://mlsanity.com/paper/2607.06483/) (2026)

## Abstract (arXiv)

This paper presents an end-to-end neural architecture based on Diffusion Models for spatial puzzle solving, particularly jigsaw puzzle and room arrangement tasks. In the latter task, for instance, the proposed system "PuzzleFusion" takes a set of room layouts as polygonal curves in the top-down view and aligns the room layout pieces by estimating their 2D translations and rotations, akin to solving the jigsaw puzzle of room layouts. A surprising discovery of the paper is that the simple use of a Diffusion Model effectively solves these challenging spatial puzzle tasks as a conditional generation process. To enable learning of an end-to-end neural system, the paper introduces new datasets with ground-truth arrangements: 1) 2D Voronoi jigsaw dataset, a synthetic one where pieces are generated by Voronoi diagram of 2D pointset; and 2) MagicPlan dataset, a real one offered by MagicPlan from its production pipeline, where pieces are room layouts constructed by augmented reality App by real-estate consumers. The qualitative and quantitative evaluations demonstrate that our approach outperforms the competing methods by significant margins in all the tasks.