# SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting

arXiv: https://arxiv.org/abs/2402.18848 · HTML: https://mlsanity.com/paper/2402.18848/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 56 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild](https://mlsanity.com/paper/1712.01261/) (2017)
- [Background Matting: The World is Your Green Screen](https://mlsanity.com/paper/2004.00626/) (2020)
- [What Do Self-Supervised Vision Transformers Learn?](https://mlsanity.com/paper/2305.00729/) (2023)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [On Efficient Transformer-Based Image Pre-training for Low-Level Vision](https://mlsanity.com/paper/2112.10175/) (2021)
- [Boundless: Generative Adversarial Networks for Image Extension](https://mlsanity.com/paper/1908.07007/) (2019)
- [Face Relighting with Geometrically Consistent Shadows](https://mlsanity.com/paper/2203.16681/) (2022)
- [Towards High Fidelity Face Relighting with Realistic Shadows](https://mlsanity.com/paper/2104.00825/) (2021)

## Built on by (most cited first, 30 of 67)

- [URAvatar: Universal Relightable Gaussian Codec Avatars](https://mlsanity.com/paper/2410.24223/) (2024)
- [LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting](https://mlsanity.com/paper/2412.00177/) (2024)
- [GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion](https://mlsanity.com/paper/2412.10209/) (2024)
- [LBM: Latent Bridge Matching for Fast Image-to-Image Translation](https://mlsanity.com/paper/2503.07535/) (2025)
- [DifFRelight: Diffusion-Based Facial Performance Relighting](https://mlsanity.com/paper/2410.08188/) (2024)
- [Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset](https://mlsanity.com/paper/2503.14485/) (2025)
- [VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation](https://mlsanity.com/paper/2502.07531/) (2025)
- [HRAvatar: High-Quality and Relightable Gaussian Head Avatar](https://mlsanity.com/paper/2503.08224/) (2025)
- [ScribbleLight: Single Image Indoor Relighting with Scribbles](https://mlsanity.com/paper/2411.17696/) (2024)
- [SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces](https://mlsanity.com/paper/2501.09756/) (2025)
- [Relightable Full-Body Gaussian Codec Avatars](https://mlsanity.com/paper/2501.14726/) (2025)
- [Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering](https://mlsanity.com/paper/2508.14461/) (2025)
- [Lumen: Consistent Video Relighting and Harmonious Background Replacement with Video Generative Models](https://mlsanity.com/paper/2508.12945/) (2025)
- [Surfel-based Gaussian Inverse Rendering for Fast and Relightable Dynamic Human Reconstruction from Monocular Video](https://mlsanity.com/paper/2407.15212/) (2024)
- [Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization](https://mlsanity.com/paper/2504.03011/) (2025)
- [UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback](https://mlsanity.com/paper/2511.01678/) (2025)
- [Light-X: Generative 4D Video Rendering with Camera and Illumination Control](https://mlsanity.com/paper/2512.05115/) (2025)
- [TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer](https://mlsanity.com/paper/2506.18904/) (2025)
- [Generative Portrait Shadow Removal](https://mlsanity.com/paper/2410.05525/) (2024)
- [LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes](https://mlsanity.com/paper/2601.15283/) (2026)
- [DreamLight: Towards Harmonious and Consistent Image Relighting](https://mlsanity.com/paper/2506.14549/) (2025)
- [IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation](https://mlsanity.com/paper/2506.03150/) (2025)
- [HumanOLAT: A Large-Scale Dataset for Full-Body Human Relighting and Novel-View Synthesis](https://mlsanity.com/paper/2508.09137/) (2025)
- [LiveLight: Real-time Streaming Video Relighting with Interactive Control](https://mlsanity.com/paper/2608.01771/) (2026)
- [Interactive Rendering of Relightable and Animatable Gaussian Avatars](https://mlsanity.com/paper/2407.10707/) (2024)
- [FlowPortal: Residual-Corrected Flow for Training-Free Video Relighting and Background Replacement](https://mlsanity.com/paper/2511.18346/) (2025)
- [Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.21943/) (2025)
- [ANYPORTAL: Zero-Shot Consistent Video Background Replacement](https://mlsanity.com/paper/2509.07472/) (2025)
- [Generalizable and Relightable Gaussian Splatting for Human Novel View Synthesis](https://mlsanity.com/paper/2505.21502/) (2025)
- [VGGTFace: Topologically Consistent Facial Geometry Reconstruction in the Wild](https://mlsanity.com/paper/2511.20366/) (2025)

## Abstract (arXiv)

We introduce a co-designed approach for human portrait relighting that combines a physics-guided architecture with a pre-training framework. Drawing on the Cook-Torrance reflectance model, we have meticulously configured the architecture design to precisely simulate light-surface interactions. Furthermore, to overcome the limitation of scarce high-quality lightstage data, we have developed a self-supervised pre-training strategy. This novel combination of accurate physical modeling and expanded training dataset establishes a new benchmark in relighting realism.