mlsanityData to 7 Oct 2026

Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting

arXiv 2503.21770 ↗First posted 27 Mar 2025cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

This paper proposes a novel scene understanding task called Visual Jenga. Drawing inspiration from the game Jenga, the proposed task involves progressively removing objects from a single image until only the background remains. Just as Jenga players must understand structural dependencies to maintain tower stability, our task reveals the intrinsic relationships between scene elements by systematically exploring which objects can be removed while preserving scene coherence in both physical and geometric sense. As a starting point for tackling the Visual Jenga task, we propose a simple, data-driven, training-free approach that is surprisingly effective on a range of real-world images. The principle behind our approach is to utilize the asymmetry in the pairwise relationships between objects within a scene and employ a large inpainting model to generate a set of counterfactuals to quantify the asymmetry.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 50 of its 89 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

50 of 89 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
38 more
  1. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  2. 2021Variational Diffusion Modelscited by 1,145
  3. 2019Panoptic Feature Pyramid Networkscited by 644
  4. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  5. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  6. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  7. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  8. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 323
  9. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  10. 2019Counterfactual Visual Explanationscited by 312
  11. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  12. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  13. 2022Visual Prompting via Image Inpaintingcited by 248
  14. 2023A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpaintingcited by 168
  15. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  16. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  17. 2020ACRONYM: A Large-Scale Grasp Dataset Based on Simulationcited by 115
  18. 2016Discovering Causal Signals in Imagescited by 79
  19. 2024ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertioncited by 71
  20. 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholescited by 70
  21. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  22. 2016Learning Physical Intuition of Block Towers by Examplecited by 66
  23. 2018Counterfactuals uncover the modular structure of deep generative modelscited by 51
  24. 2023Generative Models: What Do They Know? Do They Know Things? Let's Find Out!cited by 50
  25. 2023StyleGAN knows Normal, Depth, Albedo, and Morecited by 44
  26. 2023Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitivescited by 37
  27. 2024LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relightingcited by 37
  28. 2022BlobGAN: Spatially Disentangled Scene Representationscited by 35
  29. 2023A General Protocol to Probe Large Vision Models for 3D Physical Understandingcited by 32
  30. 2019Object-Driven Multi-Layer Scene Decomposition From a Single Imagecited by 26
  31. 2021Visiting the Invisible: Layer-by-Layer Completed Scene Decompositioncited by 23
  32. 2019How to make a pizza: Learning a compositional layer-based GAN modelcited by 21
  33. 2023Unifying (Machine) Vision via Counterfactual World Modelingcited by 18
  34. 2024Object-level Scene Deocclusioncited by 17
  35. 2016On Support Relations and Semantic Scene Graphscited by 9
  36. 2023Convex Decomposition of Indoor Scenescited by 6
  37. 2023Blocks2World: Controlling Realistic Scenes with Editable Primitivescited by 4
  38. 2024Object Remover Performance Evaluation Methods using Class-wise Object Removal Imagescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025Video models are zero-shot learners and reasonerscited by 208
  2. 2025Counterfactual Segmentation Reasoning: Diagnosing and Mitigating Pixel-Grounding Hallucinationcited by 7
  3. 2025Point Prompting: Counterfactual Tracking with Video Diffusion Modelscited by 5
  4. 2026Iterative Refinement Improves Compositional Image Generationcited by 4
  5. 2025Objects in Generated Videos Are Slower Than They Appear: Models Suffer Sub-Earth Gravity and Don't Know Galileo's Principle...for nowcited by 3
  6. 2025Generative Blocks World: Moving Things Around in Picturescited by 2
  7. 2026Multimodal Language Models Cannot Spot Spatial Inconsistenciescited by 1
  8. 2026Physical Object Understanding with a Physically Controllable World Modelcited by 1
  9. 2025Do-Undo Bench: Reversibility for Action Understanding in Image Generationcited by 0
  10. 2026CFCamo: A Counterfactual Detect-or-Abstain Framework for Camouflaged Object Detectioncited by 0
  11. 2026Principia: Relational Physics Tests for Video Modelscited by 0