# VisDiff: SDF-Guided Polygon Generation for Visibility Reconstruction and Recognition

arXiv: https://arxiv.org/abs/2410.05530 · HTML: https://mlsanity.com/paper/2410.05530/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 58 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 30 of 30)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation](https://mlsanity.com/paper/2212.04493/) (2022)
- [MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers](https://mlsanity.com/paper/2311.15475/) (2023)
- [3DGen: Triplane Latent Diffusion for Textured Mesh Generation](https://mlsanity.com/paper/2303.05371/) (2023)
- [PolyFormer: Referring Image Segmentation as Sequential Polygon Generation](https://mlsanity.com/paper/2302.07387/) (2023)
- [Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions](https://mlsanity.com/paper/2211.13757/) (2022)
- [PolyTransform: Deep Polygon Transformer for Instance Segmentation](https://mlsanity.com/paper/1912.02801/) (2019)
- [HouseDiffusion: Vector Floorplan Generation via a Diffusion Model with Discrete and Continuous Denoising](https://mlsanity.com/paper/2211.13287/) (2022)
- [PolyDiff: Generating 3D Polygonal Meshes with Diffusion Models](https://mlsanity.com/paper/2312.11417/) (2023)
- [Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries](https://mlsanity.com/paper/2211.15658/) (2022)
- [PolyDiffuse: Polygonal Shape Reconstruction via Guided Set Diffusion Models](https://mlsanity.com/paper/2306.01461/) (2023)
- [Conditional Image Generation by Conditioning Variational Auto-Encoders](https://mlsanity.com/paper/2102.12037/) (2021)
- [Is Distance Matrix Enough for Geometric Deep Learning?](https://mlsanity.com/paper/2302.05743/) (2023)
- [Interpreting and Improving Diffusion Models from an Optimization Perspective](https://mlsanity.com/paper/2306.04848/) (2023)
- [Pixel-Aligned Recurrent Queries for Multi-View 3D Object Detection](https://mlsanity.com/paper/2310.01401/) (2023)
- [PolygonGNN: Representation Learning for Polygonal Geometries with Heterogeneous Visibility Graph](https://mlsanity.com/paper/2407.00742/) (2024)
- [MGNN: Graph Neural Networks Inspired by Distance Geometry Problem](https://mlsanity.com/paper/2201.12994/) (2022)
- [Higher-Order Function Networks for Learning Composable 3D Object Representations](https://mlsanity.com/paper/1907.10388/) (2019)
- [Approximating Robot Configuration Spaces with few Convex Sets using Clique Covers of Visibility Graphs](https://mlsanity.com/paper/2310.02875/) (2023)
- [Recognizing Visibility Graphs of Polygons with Holes and Internal-External Visibility Graphs of Polygons](https://mlsanity.com/paper/1804.05105/) (2018)
- [On Visibility Graphs of Convex Fans and Terrains](https://mlsanity.com/paper/2001.06436/) (2020)

## Abstract (arXiv)

The ability to capture rich representations of combinatorial structures has enabled the application of machine learning to tasks such as analysis and generation of floorplans, terrains, images, and animations. Recent work has primarily focused on understanding structures with well-defined features, neighborhoods, or underlying distance metrics, while those lacking such characteristics remain largely unstudied. Examples of these combinatorial structures can be found in polygons, where a small change in the vertex locations causes a significant rearrangement of the combinatorial structure, expressed as a visibility or triangulation graphs. Current representation learning approaches fail to capture structures without well-defined features and distance metrics. In this paper, we study the open problem of Visibility Reconstruction: Given a visibility graph G, construct a polygon P whose visibility graph is G. We introduce VisDiff, a novel diffusion-based approach to generate polygon P from the input visibility graph G. The main novelty of our approach is that, rather than generating the polygon's vertex set directly, we first estimate the signed distance function (SDF) associated with the polygon. The SDF is then used to extract the vertex location representing the final polygon. We show that going through the SDF allows VisDiff to learn the visibility relationship much more effectively than generating vertex locations directly. In order to train VisDiff, we create a carefully curated dataset. We use this dataset to benchmark our method and achieve 26% improvement in F1-Score over standard methods as well as state of the art approaches.