# SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches

arXiv: https://arxiv.org/abs/2502.07556 · HTML: https://mlsanity.com/paper/2502.07556/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 60 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 30 of 38)

- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs](https://mlsanity.com/paper/2401.11708/) (2024)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [LCM-LoRA: A Universal Stable-Diffusion Acceleration Module](https://mlsanity.com/paper/2311.05556/) (2023)
- [InstanceDiffusion: Instance-level Control for Image Generation](https://mlsanity.com/paper/2402.03290/) (2024)
- [Dense Text-to-Image Generation with Attention Modulation](https://mlsanity.com/paper/2308.12964/) (2023)
- [DOCCI: Descriptions of Connected and Contrasting Images](https://mlsanity.com/paper/2404.19753/) (2024)
- [Learning to Predict Visual Attributes in the Wild](https://mlsanity.com/paper/2106.09707/) (2021)
- [Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models](https://mlsanity.com/paper/2304.09337/) (2023)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works](https://mlsanity.com/paper/2210.08477/) (2022)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- [StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation](https://mlsanity.com/paper/2312.12491/) (2023)
- [Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering](https://mlsanity.com/paper/2303.13534/) (2023)
- [3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows](https://mlsanity.com/paper/2210.11603/) (2022)
- [Picture that Sketch: Photorealistic Image Generation from Abstract Sketches](https://mlsanity.com/paper/2303.11162/) (2023)
- [PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions](https://mlsanity.com/paper/2308.05184/) (2023)
- [On Discrete Prompt Optimization for Diffusion Models](https://mlsanity.com/paper/2407.01606/) (2024)

## Built on by (most cited first, 6 of 6)

- [GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design](https://mlsanity.com/paper/2508.15227/) (2025)
- [MagicQuillV2: Precise and Interactive Image Editing with Layered Visual Cues](https://mlsanity.com/paper/2512.03046/) (2025)
- [AnimAgents: Coordinating Multi-Stage Animation Pre-Production with Human-Multi-Agent Collaboration](https://mlsanity.com/paper/2511.17906/) (2025)
- [WordCraft: Scaffolding the Keyword Method for L2 Vocabulary Learning with Multimodal LLMs](https://mlsanity.com/paper/2602.00762/) (2026)
- [Adaptive Prompt Elicitation for Text-to-Image Generation](https://mlsanity.com/paper/2602.04713/) (2026)
- [From Style Replication to Style Exploration: Enabling Art Style Exploration with Analyze-Experiment-Resituate Framework](https://mlsanity.com/paper/2608.14405/) (2026)

## Abstract (arXiv)

Text-to-image models can generate visually appealing images from text descriptions. Efforts have been devoted to improving model controls with prompt tuning and spatial conditioning. However, our formative study highlights the challenges for non-expert users in crafting appropriate prompts and specifying fine-grained spatial conditions (e.g., depth or canny references) to generate semantically cohesive images, especially when multiple objects are involved. In response, we introduce SketchFlex, an interactive system designed to improve the flexibility of spatially conditioned image generation using rough region sketches. The system automatically infers user prompts with rational descriptions within a semantic space enriched by crowd-sourced object attributes and relationships. Additionally, SketchFlex refines users' rough sketches into canny-based shape anchors, ensuring the generation quality and alignment of user intentions. Experimental results demonstrate that SketchFlex achieves more cohesive image generations than end-to-end models, meanwhile significantly reducing cognitive load and better matching user intentions compared to region-based generation baseline.