# SePaint: Semantic Map Inpainting via Multinomial Diffusion

arXiv: https://arxiv.org/abs/2303.02737 · HTML: https://mlsanity.com/paper/2303.02737/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 35 references that are in the map.
- First posted on arXiv 5 Mar 2023.

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning](https://mlsanity.com/paper/2208.04202/) (2022)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [Continuous diffusion for categorical data](https://mlsanity.com/paper/2211.15089/) (2022)
- [Dual Diffusion Implicit Bridges for Image-to-Image Translation](https://mlsanity.com/paper/2203.08382/) (2022)
- [Score-based Continuous-time Discrete Diffusion Models](https://mlsanity.com/paper/2211.16750/) (2022)
- [A* Sampling](https://mlsanity.com/paper/1411.0030/) (2014)
- [Self-conditioned Embedding Diffusion for Text Generation](https://mlsanity.com/paper/2211.04236/) (2022)
- [Occupancy Anticipation for Efficient Exploration and Navigation](https://mlsanity.com/paper/2008.09285/) (2020)
- [Learning-based 3D Occupancy Prediction for Autonomous Navigation in Occluded Environments](https://mlsanity.com/paper/2011.03981/) (2020)
- [Map-Predictive Motion Planning in Unknown Environments](https://mlsanity.com/paper/1910.08184/) (2019)

## Built on by (most cited first, 1 of 1)

- [Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives](https://mlsanity.com/paper/2404.08926/) (2024)

## Abstract (arXiv)

Prediction beyond partial observations is crucial for robots to navigate in unknown environments because it can provide extra information regarding the surroundings beyond the current sensing range or resolution. In this work, we consider the inpainting of semantic Bird's-Eye-View maps. We propose SePaint, an inpainting model for semantic data based on generative multinomial diffusion. To maintain semantic consistency, we need to condition the prediction for the missing regions on the known regions. We propose a novel and efficient condition strategy, Look-Back Condition (LB-Con), which performs one-step look-back operations during the reverse diffusion process. By doing so, we are able to strengthen the harmonization between unknown and known parts, leading to better completion performance. We have conducted extensive experiments on different datasets, showing our proposed model outperforms commonly used interpolation methods in various robotic applications.