# Text2Relight: Creative Portrait Relighting with Text Guidance

arXiv: https://arxiv.org/abs/2412.13734 · HTML: https://mlsanity.com/paper/2412.13734/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation](https://mlsanity.com/paper/2209.10510/) (2022)
- [Relightful Harmonization: Lighting-aware Portrait Background Replacement](https://mlsanity.com/paper/2312.06886/) (2023)
- [Neural Video Portrait Relighting in Real-time via Consistency Modeling](https://mlsanity.com/paper/2104.00484/) (2021)
- [Face Relighting with Geometrically Consistent Shadows](https://mlsanity.com/paper/2203.16681/) (2022)
- [Towards High Fidelity Face Relighting with Realistic Shadows](https://mlsanity.com/paper/2104.00825/) (2021)
- [Geometry-aware Single-image Full-body Human Relighting](https://mlsanity.com/paper/2207.04750/) (2022)
- [Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image](https://mlsanity.com/paper/2403.09632/) (2024)
- [SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage](https://mlsanity.com/paper/2204.03648/) (2022)
- [Half-body Portrait Relighting with Overcomplete Lighting Representation](https://mlsanity.com/paper/2106.13425/) (2021)

## Built on by (most cited first, 7 of 7)

- [Lumen: Consistent Video Relighting and Harmonious Background Replacement with Video Generative Models](https://mlsanity.com/paper/2508.12945/) (2025)
- [Light-X: Generative 4D Video Rendering with Camera and Illumination Control](https://mlsanity.com/paper/2512.05115/) (2025)
- [Parametric Shadow Control for Portrait Generation in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.21943/) (2025)
- [ELIQ: A Label-Free Framework for Quality Assessment of Evolving AI-Generated Images](https://mlsanity.com/paper/2602.03558/) (2026)
- [TokenLight: Precise Lighting Control in Images using Attribute Tokens](https://mlsanity.com/paper/2604.15310/) (2026)
- [HarmoVid: Relightful Video Portrait Harmonization](https://mlsanity.com/paper/2605.28811/) (2026)
- [WarpI2I: Image Warping for Image-to-Image Translation](https://mlsanity.com/paper/2606.31018/) (2026)

## Abstract (arXiv)

We present a lighting-aware image editing pipeline that, given a portrait image and a text prompt, performs single image relighting. Our model modifies the lighting and color of both the foreground and background to align with the provided text description. The unbounded nature in creativeness of a text allows us to describe the lighting of a scene with any sensory features including temperature, emotion, smell, time, and so on. However, the modeling of such mapping between the unbounded text and lighting is extremely challenging due to the lack of dataset where there exists no scalable data that provides large pairs of text and relighting, and therefore, current text-driven image editing models does not generalize to lighting-specific use cases. We overcome this problem by introducing a novel data synthesis pipeline: First, diverse and creative text prompts that describe the scenes with various lighting are automatically generated under a crafted hierarchy using a large language model (*e.g.,* ChatGPT). A text-guided image generation model creates a lighting image that best matches the text. As a condition of the lighting images, we perform image-based relighting for both foreground and background using a single portrait image or a set of OLAT (One-Light-at-A-Time) images captured from lightstage system. Particularly for the background relighting, we represent the lighting image as a set of point lights and transfer them to other background images. A generative diffusion model learns the synthesized large-scale data with auxiliary task augmentation (*e.g.,* portrait delighting and light positioning) to correlate the latent text and lighting distribution for text-guided portrait relighting.