mlsanityData to 7 Oct 2026

pix2gestalt: Amodal Segmentation by Synthesizing Wholes

arXiv 2401.14398 ↗First posted 25 Jan 2024cs.CV cs.LGcited by 70

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

We introduce pix2gestalt, a framework for zero-shot amodal segmentation, which learns to estimate the shape and appearance of whole objects that are only partially visible behind occlusions. By capitalizing on large-scale diffusion models and transferring their representations to this task, we learn a conditional diffusion model for reconstructing whole objects in challenging zero-shot cases, including examples that break natural and physical priors, such as art. As training data, we use a synthetically curated dataset containing occluded objects paired with their whole counterparts. Experiments show that our approach outperforms supervised baselines on established benchmarks. Our model can furthermore be used to significantly improve the performance of existing object recognition and 3D reconstruction methods in the presence of occlusions.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 39 of its 56 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

39 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models8 Mar
  2. 2021Label-Efficient Semantic Segmentation with Diffusion Models6 Dec
  3. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Models1 Dec

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2023Segment Anythingcited by 8,158
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  12. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
24 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  5. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  6. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  7. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  8. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  9. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  10. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
  11. 2021Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayerscited by 104
  12. 2015Semantic Amodal Segmentationcited by 102
  13. 2020Self-Supervised Scene De-occlusioncited by 96
  14. 2017SeGAN: Segmenting and Generating the Invisiblecited by 60
  15. 2023Amodal Ground Truth and Completion in the Wildcited by 43
  16. 2023Amodal Completion via Progressive Mixed Context Diffusioncited by 38
  17. 2023Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shapecited by 29
  18. 20233D-Aware Neural Body Fitting for Occlusion Robust 3D Human Pose Estimationcited by 28
  19. 2022Robust Category-Level 6D Pose Estimation with Coarse-to-Fine Rendering of Neural Featurescited by 23
  20. 2015Amodal Completion and Size Constancy in Natural Scenescited by 21
  21. 2022A Tri-Layer Plugin to Improve Occluded Detectioncited by 21
  22. 2022Landscape Learning for Neural Network Inversioncited by 15
  23. 2023Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflectioncited by 10
  24. 2022Shadows Shed Light on 3D Objectscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 30; 2026: 22.

70 papers.

Sort:

By area:

  1. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 116
  2. 2024Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single Viewcited by 59
  3. 2025HoloPart: Generative 3D Part Amodal Segmentationcited by 57
  4. 2025Amodal3R: Amodal 3D Reconstruction from Occluded 2D Imagescited by 52
  5. 2025GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Trackingcited by 37
  6. 2025ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediarycited by 27
  7. 2024Controlling the World by Sleight of Handcited by 19
  8. 2024Open-World Amodal Appearance Completioncited by 18
  9. 2025HiScene: Creating Hierarchical 3D Scenes with Isometric View Generationcited by 18
  10. 2024Object-level Scene Deocclusioncited by 17
  11. 2025R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulationcited by 17
  12. 2024Scaling Properties of Diffusion Models for Perceptual Taskscited by 15
38 more
  1. 2024Using Diffusion Priors for Video Amodal Segmentationcited by 15
  2. 2026Walk through Paintings: Egocentric World Models from Internet Priorscited by 13
  3. 2025SceneMI: Motion In-betweening for Modeling Human-Scene Interactionscited by 11
  4. 2025Visual Jenga: Discovering Object Dependencies via Counterfactual Inpaintingcited by 11
  5. 2024ReferEverything: Towards Segmenting Everything We Can Speak of in Videoscited by 10
  6. 2025TACO: Taming Diffusion for in-the-wild Video Amodal Completioncited by 10
  7. 2025DeOcc-1-to-3: 3D De-Occlusion from a Single Image via Self-Supervised Multi-View Diffusioncited by 10
  8. 2024MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenescited by 9
  9. 2025Beyond the Visible: Benchmarking Occlusion Perception in Multimodal Large Language Modelscited by 8
  10. 2024Robust 3D Shape Reconstruction in Zero-Shot from a Single Image in the Wildcited by 7
  11. 2025Segment Anything, Even Occludedcited by 7
  12. 2024Inverse Painting: Reconstructing The Painting Processcited by 6
  13. 2024Amodal Instance Segmentation with Diffusion Shape Prior Estimationcited by 5
  14. 2025A Diffusion-Based Framework for Occluded Object Movementcited by 5
  15. 2024EraseDraw: Learning to Draw Step-by-Step via Erasing Objects from Imagescited by 4
  16. 2024Amodal Depth Anything: Amodal Depth Estimation in the Wildcited by 4
  17. 2025GENA3D: Generative Amodal 3D Modeling by Bridging 2D Priors and 3D Coherencecited by 4
  18. 2026I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editingcited by 4
  19. 2024TARGO: Benchmarking Target-driven Object Grasping under Occlusionscited by 3
  20. 2025PatchRefiner V2: Fast and Lightweight Real-Domain High-Resolution Metric Depth Estimationcited by 3
  21. 2025Unlocking Constraints: Source-Free Occlusion-Aware Seamless Segmentationcited by 3
  22. 2025Efficient and Scalable Monocular Human-Object Interaction Motion Reconstructioncited by 3
  23. 2026Referring Layer Decompositioncited by 3
  24. 2026HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehiclescited by 3
  25. 2026AmodalSVG: Amodal Image Vectorization via Semantic Layer Peelingcited by 3
  26. 2025CompleteMe: Reference-based Human Image Completioncited by 2
  27. 2025gen2seg: Generative Models Enable Generalizable Instance Segmentationcited by 2
  28. 2025DreamAnywhere: Object-Centric Panoramic 3D Scene Generationcited by 2
  29. 2026Amodal SAM: A Unified Amodal Segmentation Framework with Generalizationcited by 2
  30. 2026Axolotl3D: a Unified Framework for Faithful 3D Shape Completioncited by 2
  31. 2026WorldSculpt: Generating Compositional Worlds from Grounded Videoscited by 2
  32. 2025EscherNet++: Simultaneous Amodal Completion and Scalable View Synthesis through Masked Fine-Tuning and Enhanced Feed-Forward 3D Reconstructioncited by 1
  33. 2025Contact-Aware Amodal Completion for Human-Object Interaction via Multi-Regional Inpaintingcited by 1
  34. 2025Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusionscited by 1
  35. 2025CObL: Toward Zero-Shot Ordinal Layering without User Promptingcited by 1
  36. 2025Object Reconstruction under Occlusion with Generative Priors and Contact-induced Constraintscited by 1
  37. 2025Reasoning-Driven Amodal Completion: Collaborative Agents and Perceptual Evaluationcited by 1
  38. 2026Seeing Through Clutter: Structured 3D Scene Reconstruction via Iterative Object Removalcited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Sequential Amodal Segmentation via Cumulative Occlusion Learning
  2. 2025Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation
  3. 2024Part123: Part-aware 3D Reconstruction from a Single-view Image
  4. 2024Diffusion Models in 3D Vision: A Survey
  5. 2025Studying Image Diffusion Features for Zero-Shot Video Object Segmentation
  6. 2024GRIN: Zero-Shot Metric Depth with Pixel-Level Diffusion
  7. 2026Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentation
  8. 2026MAVISEG: Manifold Propagation and Visual Prototypes for Zero-Shot Open-Vocabulary Segmentation in Diffusion Transformers