# Physically-guided Image Generation for Multi-Projection Mapping

arXiv: https://arxiv.org/abs/2606.22477 · HTML: https://mlsanity.com/paper/2606.22477/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 56 references that are in the map.
- First posted on arXiv 21 Jun 2026.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers](https://mlsanity.com/paper/2410.10629/) (2024)
- [ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback](https://mlsanity.com/paper/2404.07987/) (2024)
- [RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination](https://mlsanity.com/paper/2505.21925/) (2025)
- [End-to-end Full Projector Compensation](https://mlsanity.com/paper/2008.00965/) (2020)
- [End-to-end Projector Photometric Compensation](https://mlsanity.com/paper/1904.04335/) (2019)
- [Neural Projection Mapping Using Reflectance Fields](https://mlsanity.com/paper/2306.06595/) (2023)
- [CompenNet++: End-to-end Full Projector Compensation](https://mlsanity.com/paper/1908.06246/) (2019)
- [Casper DPM: Cascaded Perceptual Dynamic Projection Mapping onto Hands](https://mlsanity.com/paper/2409.04397/) (2024)
- [CompenHR: Efficient Full Compensation for High-resolution Projector](https://mlsanity.com/paper/2311.13409/) (2023)
- [GS-ProCams: Gaussian Splatting-based Projector-Camera Systems](https://mlsanity.com/paper/2412.11762/) (2024)
- [LAPIG: Language Guided Projector Image Generation with Surface Adaptation and Stylization](https://mlsanity.com/paper/2503.12173/) (2025)
- [Robust Single-shot Structured Light 3D Imaging via Neural Feature Decoding](https://mlsanity.com/paper/2512.14028/) (2025)

## Abstract (arXiv)

Projection Mapping (PM) enables seamless superimposition of digital content onto real-world 3D objects, serving as a fundamental technique for immersive visualization, digital twins, and interactive art. Although text-to-image diffusion models have greatly facilitated customized content creation, directly integrating them into practical PM pipelines remains challenging due to the mismatch between idealized 2D generation and physical constraints. To bridge this gap, this paper formalizes two application-level generative paradigms: the cooperative paradigm (harmonizing generated semantics with physical attributes) and the adversarial paradigm (eliminating surface interference via radiometric compensation). Based on this, we propose ConPhyG, a unified controllable physically-guided generative multi-projection mapping framework that enables creators to interactively adjust physical constraints and flexibly switch generative paradigms. In cooperative mode, multi-dimensional physical priors (per-pixel gamut, depth, and edges) are injected into the diffusion process. In adversarial mode, the framework releases the generative potential and applies bounded numerical optimization for multi-projector radiometric compensation. It allows users to dynamically switch constraints to balance artistic freedom with physical feasibility. Furthermore, we extend ConPhyG to 360-degree multi-view consistent PM using a sequential generation strategy. Quantitative and qualitative evaluations on a real-world four-projector setup demonstrate that ConPhyG significantly outperforms state-of-the-art methods in geometric alignment, gamut utilization, and semantic fidelity.