# Replace in Translation: Boost Concept Alignment in Counterfactual Text-to-Image

arXiv: https://arxiv.org/abs/2505.14341 · HTML: https://mlsanity.com/paper/2505.14341/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 28 of 28)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers](https://mlsanity.com/paper/2410.10629/) (2024)
- [MirrorGAN: Learning Text-to-image Generation by Redescription](https://mlsanity.com/paper/1903.05854/) (2019)
- [Rich Human Feedback for Text-to-Image Generation](https://mlsanity.com/paper/2312.10240/) (2023)
- [Generating Images from Captions with Attention](https://mlsanity.com/paper/1511.02793/) (2015)
- [Counterfactual Zero-Shot and Open-Set Visual Recognition](https://mlsanity.com/paper/2103.00887/) (2021)
- [Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2405.00760/) (2024)
- [Highly Personalized Text Embedding for Image Manipulation by Stable Diffusion](https://mlsanity.com/paper/2303.08767/) (2023)
- [Text-to-Image Generation Grounded by Fine-Grained User Attention](https://mlsanity.com/paper/2011.03775/) (2020)
- [Customizing Text-to-Image Models with a Single Image Pair](https://mlsanity.com/paper/2405.01536/) (2024)
- [ReFACT: Updating Text-to-Image Models by Editing the Text Encoder](https://mlsanity.com/paper/2306.00738/) (2023)
- [Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models](https://mlsanity.com/paper/2310.07653/) (2023)
- [AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generation](https://mlsanity.com/paper/2406.18958/) (2024)
- [A Data Perspective on Enhanced Identity Preservation for Diffusion Personalization](https://mlsanity.com/paper/2311.04315/) (2023)
- [DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Design](https://mlsanity.com/paper/2310.15144/) (2023)
- [Training-Free Sketch-Guided Diffusion with Latent Optimization](https://mlsanity.com/paper/2409.00313/) (2024)
- [ACTesting: Automated Cross-modal Testing Method of Text-to-Image Software](https://mlsanity.com/paper/2312.12933/) (2023)

## Built on by (most cited first, 1 of 1)

- [Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning](https://mlsanity.com/paper/2606.24548/) (2026)

## Abstract (arXiv)

Text-to-Image (T2I) has been prevalent in recent years, with most common condition tasks having been optimized nicely. Besides, counterfactual Text-to-Image is obstructing us from a more versatile AIGC experience. For those scenes that are impossible to happen in real world and anti-physics, we should spare no efforts in increasing the factual feel, which means synthesizing images that people think very likely to be happening, and concept alignment, which means all the required objects should be in the same frame. In this paper, we focus on concept alignment. As controllable T2I models have achieved satisfactory performance for real applications, we utilize this technology to replace the objects in a synthesized image in latent space step-by-step to change the image from a common scene to a counterfactual scene to meet the prompt. We propose a strategy to instruct this replacing process, which is called as Explicit Logical Narrative Prompt (ELNP), by using the newly SoTA language model DeepSeek to generate the instructions. Furthermore, to evaluate models' performance in counterfactual T2I, we design a metric to calculate how many required concepts in the prompt can be covered averagely in the synthesized images. The extensive experiments and qualitative comparisons demonstrate that our strategy can boost the concept alignment in counterfactual T2I.