# SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing

arXiv: https://arxiv.org/abs/2312.11392 · HTML: https://mlsanity.com/paper/2312.11392/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 35 of its 49 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 35)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [Composer: Creative and Controllable Image Synthesis with Composable Conditions](https://mlsanity.com/paper/2302.09778/) (2023)
- [UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild](https://mlsanity.com/paper/2305.11147/) (2023)
- [FreeU: Free Lunch in Diffusion U-Net](https://mlsanity.com/paper/2309.11497/) (2023)

## Built on by (most cited first, 30 of 37)

- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling](https://mlsanity.com/paper/2501.02487/) (2025)
- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion](https://mlsanity.com/paper/2410.19324/) (2024)
- [Controlling Language and Diffusion Models by Transporting Activations](https://mlsanity.com/paper/2410.23054/) (2024)
- [Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidance](https://mlsanity.com/paper/2412.12974/) (2024)
- [ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer](https://mlsanity.com/paper/2410.00086/) (2024)
- [UniRestore: Unified Perceptual and Task-Oriented Image Restoration Model Using Diffusion Prior](https://mlsanity.com/paper/2501.13134/) (2025)
- [Conditional Image Synthesis with Diffusion Models: A Survey](https://mlsanity.com/paper/2409.19365/) (2024)
- [StyleBooth: Image Style Editing with Multimodal Instruction](https://mlsanity.com/paper/2404.12154/) (2024)
- [ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editing](https://mlsanity.com/paper/2503.14482/) (2025)
- [CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation](https://mlsanity.com/paper/2410.09400/) (2024)
- [From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation](https://mlsanity.com/paper/2404.15267/) (2024)
- [The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling](https://mlsanity.com/paper/2402.15170/) (2024)
- [Diffusion Model Patching via Mixture-of-Prompts](https://mlsanity.com/paper/2405.17825/) (2024)
- [Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearning](https://mlsanity.com/paper/2507.07139/) (2025)
- [SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation](https://mlsanity.com/paper/2511.16666/) (2025)
- [BokehDiff: Neural Lens Blur with One-Step Diffusion](https://mlsanity.com/paper/2507.18060/) (2025)
- [VISTA: Triplet-Supervised Video Style Transfer with Diffusion Transformers](https://mlsanity.com/paper/2605.17312/) (2026)
- [CasSR: Activating Image Power for Real-World Image Super-Resolution](https://mlsanity.com/paper/2403.11451/) (2024)
- [KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Models](https://mlsanity.com/paper/2410.01595/) (2024)
- [Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Model](https://mlsanity.com/paper/2506.15682/) (2025)
- [NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generation](https://mlsanity.com/paper/2512.05106/) (2025)
- [Locate, Assign, Refine: Taming Customized Promptable Image Inpainting](https://mlsanity.com/paper/2403.19534/) (2024)
- [CTRLorALTer: Conditional LoRAdapter for Efficient 0-Shot Control & Altering of T2I Models](https://mlsanity.com/paper/2405.07913/) (2024)
- [Exploring Probabilistic Modeling Beyond Domain Generalization for Semantic Segmentation](https://mlsanity.com/paper/2507.21367/) (2025)
- [A3D: Does Diffusion Dream about 3D Alignment?](https://mlsanity.com/paper/2406.15020/) (2024)
- [Accelerate High-Quality Diffusion Models with Inner Loop Feedback](https://mlsanity.com/paper/2501.13107/) (2025)
- [Avatar Concept Slider: Controllable Editing of Concepts in 3D Human Avatars](https://mlsanity.com/paper/2408.13995/) (2024)

## Abstract (arXiv)

Image diffusion models have been utilized in various tasks, such as text-to-image generation and controllable image synthesis. Recent research has introduced tuning methods that make subtle adjustments to the original models, yielding promising results in specific adaptations of foundational generative diffusion models. Rather than modifying the main backbone of the diffusion model, we delve into the role of skip connection in U-Net and reveal that hierarchical features aggregating long-distance information across encoder and decoder make a significant impact on the content and quality of image generation. Based on the observation, we propose an efficient generative tuning framework, dubbed SCEdit, which integrates and edits Skip Connection using a lightweight tuning module named SC-Tuner. Furthermore, the proposed framework allows for straightforward extension to controllable image synthesis by injecting different conditions with Controllable SC-Tuner, simplifying and unifying the network design for multi-condition inputs. Our SCEdit substantially reduces training parameters, memory usage, and computational expense due to its lightweight tuners, with backward propagation only passing to the decoder blocks. Extensive experiments conducted on text-to-image generation and controllable image synthesis tasks demonstrate the superiority of our method in terms of efficiency and performance. Project page: urlhttps://scedit.github.io/