# Learning by Planning: Language-Guided Global Image Editing

arXiv: https://arxiv.org/abs/2106.13156 · HTML: https://mlsanity.com/paper/2106.13156/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding](https://mlsanity.com/paper/1810.02338/) (2018)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [Kornia: an Open Source Differentiable Computer Vision Library for PyTorch](https://mlsanity.com/paper/1910.02190/) (2019)
- [UPSNet: A Unified Panoptic Segmentation Network](https://mlsanity.com/paper/1901.03784/) (2019)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [ManiGAN: Text-Guided Image Manipulation](https://mlsanity.com/paper/1912.06203/) (2019)
- [Learning to Assemble Neural Module Tree Networks for Visual Grounding](https://mlsanity.com/paper/1812.03299/) (2018)
- [Inferring and Executing Programs for Visual Reasoning](https://mlsanity.com/paper/1705.03633/) (2017)
- [Learning to Reason: End-to-End Module Networks for Visual Question Answering](https://mlsanity.com/paper/1704.05526/) (2017)
- [Modeling Relationships in Referential Expressions with Compositional Modular Networks](https://mlsanity.com/paper/1611.09978/) (2016)
- [Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language](https://mlsanity.com/paper/1810.11919/) (2018)
- [Semantic Image Synthesis via Adversarial Learning](https://mlsanity.com/paper/1707.06873/) (2017)
- [Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction](https://mlsanity.com/paper/1811.09845/) (2018)
- [Diverse Image Synthesis from Semantic Layouts via Conditional IMLE](https://mlsanity.com/paper/1811.12373/) (2018)
- [Explainable Neural Computation via Stack Neural Module Networks](https://mlsanity.com/paper/1807.08556/) (2018)
- [Sequential Attention GAN for Interactive Image Editing](https://mlsanity.com/paper/1812.08352/) (2018)
- [Exposure: A White-Box Photo Post-Processing Framework](https://mlsanity.com/paper/1709.09602/) (2017)
- [A Benchmark and Baseline for Language-Driven Image Editing](https://mlsanity.com/paper/2010.02330/) (2020)
- [Bilinear Representation for Language-based Image Editing Using Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1903.07499/) (2019)

## Built on by (most cited first, 25 of 25)

- [Guiding Instruction-based Image Editing via Multimodal Large Language Models](https://mlsanity.com/paper/2309.17102/) (2023)
- [Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editing](https://mlsanity.com/paper/2510.19808/) (2025)
- [DCCF: Deep Comprehensible Color Filter Learning Framework for High-Resolution Image Harmonization](https://mlsanity.com/paper/2207.04788/) (2022)
- [Towards Semantic Equivalence of Tokenization in Multimodal LLM](https://mlsanity.com/paper/2406.05127/) (2024)
- [Auto-Encoding Morph-Tokens for Multimodal LLM](https://mlsanity.com/paper/2405.01926/) (2024)
- [EmoEdit: Evoking Emotions through Image Manipulation](https://mlsanity.com/paper/2405.12661/) (2024)
- [HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editing](https://mlsanity.com/paper/2412.04280/) (2024)
- [Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens](https://mlsanity.com/paper/2504.14666/) (2025)
- [Towards Generic Image Manipulation Detection with Weakly-Supervised Self-Consistency Learning](https://mlsanity.com/paper/2309.01246/) (2023)
- [CLIPtone: Unsupervised Learning for Text-based Image Tone Adjustment](https://mlsanity.com/paper/2404.01123/) (2024)
- [MonetGPT: Solving Puzzles Enhances MLLMs' Image Retouching Skills](https://mlsanity.com/paper/2505.06176/) (2025)
- [Neural Color Operators for Sequential Image Retouching](https://mlsanity.com/paper/2207.08080/) (2022)
- [Text-to-Image Cross-Modal Generation: A Systematic Review](https://mlsanity.com/paper/2401.11631/) (2024)
- [Understanding Generative AI Capabilities in Everyday Image Editing Tasks](https://mlsanity.com/paper/2505.16181/) (2025)
- [RetouchIQ: MLLM Agents for Instruction-Based Image Retouching with Generalist Reward](https://mlsanity.com/paper/2602.17558/) (2026)
- [Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning](https://mlsanity.com/paper/2306.11065/) (2023)
- [SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing](https://mlsanity.com/paper/2112.00180/) (2021)
- [Contrastive Monotonic Pixel-Level Modulation](https://mlsanity.com/paper/2207.11517/) (2022)
- [Image Manipulation via Multi-Hop Instructions -- A New Dataset and Weakly-Supervised Neuro-Symbolic Approach](https://mlsanity.com/paper/2305.14410/) (2023)
- [What Changed? Detecting and Evaluating Instruction-Guided Image Edits with Multimodal Large Language Models](https://mlsanity.com/paper/2505.20405/) (2025)
- [Language-driven Fine-grained Retrieval](https://mlsanity.com/paper/2512.06255/) (2025)
- [IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment](https://mlsanity.com/paper/2606.08016/) (2026)
- [Interactive Neural Painting](https://mlsanity.com/paper/2307.16441/) (2023)
- [UIP2P: Unsupervised Instruction-based Image Editing via Edit Reversibility Constraint](https://mlsanity.com/paper/2412.15216/) (2024)
- [Cascade Luminance and Chrominance for Image Retouching: More Like Artist](https://mlsanity.com/paper/2205.15999/) (2022)

## Abstract (arXiv)

Recently, language-guided global image editing draws increasing attention with growing application potentials. However, previous GAN-based methods are not only confined to domain-specific, low-resolution data but also lacking in interpretability. To overcome the collective difficulties, we develop a text-to-operation model to map the vague editing language request into a series of editing operations, e.g., change contrast, brightness, and saturation. Each operation is interpretable and differentiable. Furthermore, the only supervision in the task is the target image, which is insufficient for a stable training of sequential decisions. Hence, we propose a novel operation planning algorithm to generate possible editing sequences from the target image as pseudo ground truth. Comparison experiments on the newly collected MA5k-Req dataset and GIER dataset show the advantages of our methods. Code is available at https://jshi31.github.io/T2ONet.