# SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning

arXiv: https://arxiv.org/abs/2009.09566 · HTML: https://mlsanity.com/paper/2009.09566/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Sep 2020.

## Built on by (most cited first, 30 of 32)

- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [ImgEdit: A Unified Image Editing Dataset and Benchmark](https://mlsanity.com/paper/2505.20275/) (2025)
- [Guiding Instruction-based Image Editing via Multimodal Large Language Models](https://mlsanity.com/paper/2309.17102/) (2023)
- [Talk-to-Edit: Fine-Grained Facial Editing via Dialog](https://mlsanity.com/paper/2109.04425/) (2021)
- [Language-Driven Image Style Transfer](https://mlsanity.com/paper/2106.00178/) (2021)
- [Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation](https://mlsanity.com/paper/2211.12824/) (2022)
- [InstructX: Towards Unified Visual Editing with MLLM Guidance](https://mlsanity.com/paper/2510.08485/) (2025)
- [Iterative Multi-granular Image Editing using Diffusion Models](https://mlsanity.com/paper/2309.00613/) (2023)
- [TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation](https://mlsanity.com/paper/2404.18919/) (2024)
- [ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generation](https://mlsanity.com/paper/2204.04428/) (2022)
- [Learning to Follow Object-Centric Image Editing Instructions Faithfully](https://mlsanity.com/paper/2310.19145/) (2023)
- [Element Intervention for Open Relation Extraction](https://mlsanity.com/paper/2106.09558/) (2021)
- [Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset](https://mlsanity.com/paper/2302.14406/) (2023)
- [M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers](https://mlsanity.com/paper/2104.01122/) (2021)
- [Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialogue](https://mlsanity.com/paper/2106.15550/) (2021)
- [InstructRL4Pix: Training Diffusion for Image Editing by Reinforcement Learning](https://mlsanity.com/paper/2406.09973/) (2024)
- [Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learning](https://mlsanity.com/paper/2512.09924/) (2025)
- [Repositioning the Subject within Image](https://mlsanity.com/paper/2401.16861/) (2024)
- [LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulation](https://mlsanity.com/paper/2112.13985/) (2021)
- [LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-training](https://mlsanity.com/paper/2201.08081/) (2022)
- [A Diffusion-based Method for Multi-turn Compositional Image Generation](https://mlsanity.com/paper/2304.02192/) (2023)
- [Text-guided 3D Human Generation from 2D Collections](https://mlsanity.com/paper/2305.14312/) (2023)
- [CAISE: Conversational Agent for Image Search and Editing](https://mlsanity.com/paper/2202.11847/) (2022)
- [Entity-Level Text-Guided Image Manipulation](https://mlsanity.com/paper/2302.11383/) (2023)
- [Video Generation Beyond a Single Clip](https://mlsanity.com/paper/2304.07483/) (2023)
- [Learning Action-Effect Dynamics for Hypothetical Vision-Language Reasoning Task](https://mlsanity.com/paper/2212.03866/) (2022)
- [Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA](https://mlsanity.com/paper/2305.19664/) (2023)
- [Diffusion idea exploration for art generation](https://mlsanity.com/paper/2307.04978/) (2023)

## Abstract (arXiv)

Iterative Language-Based Image Editing (IL-BIE) tasks follow iterative instructions to edit images step by step. Data scarcity is a significant issue for ILBIE as it is challenging to collect large-scale examples of images before and after instruction-based changes. However, humans still accomplish these editing tasks even when presented with an unfamiliar image-instruction pair. Such ability results from counterfactual thinking and the ability to think about alternatives to events that have happened already. In this paper, we introduce a Self-Supervised Counterfactual Reasoning (SSCR) framework that incorporates counterfactual thinking to overcome data scarcity. SSCR allows the model to consider out-of-distribution instructions paired with previous images. With the help of cross-task consistency (CTC), we train these counterfactual instructions in a self-supervised scenario. Extensive results show that SSCR improves the correctness of ILBIE in terms of both object identity and position, establishing a new state of the art (SOTA) on two IBLIE datasets (i-CLEVR and CoDraw). Even with only 50% of the training data, SSCR achieves a comparable result to using complete data.