# MagicStick: Controllable Video Editing via Control Handle Transformations

arXiv: https://arxiv.org/abs/2312.03047 · HTML: https://mlsanity.com/paper/2312.03047/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 92 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 30 of 68)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)

## Built on by (most cited first, 30 of 62)

- [Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation](https://mlsanity.com/paper/2406.01900/) (2024)
- [Identity-Preserving Text-to-Video Generation by Frequency Decomposition](https://mlsanity.com/paper/2411.17440/) (2024)
- [DiT4Edit: Diffusion Transformer for Image Editing](https://mlsanity.com/paper/2411.03286/) (2024)
- [Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models](https://mlsanity.com/paper/2503.20752/) (2025)
- [Controllable Video Generation: A Survey](https://mlsanity.com/paper/2507.16869/) (2025)
- [Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuning](https://mlsanity.com/paper/2506.05207/) (2025)
- [Follow-Your-Creation: Empowering 4D Creation through Video Inpainting](https://mlsanity.com/paper/2506.04590/) (2025)
- [Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives](https://mlsanity.com/paper/2403.10518/) (2024)
- [COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing](https://mlsanity.com/paper/2406.08850/) (2024)
- [Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts](https://mlsanity.com/paper/2403.08268/) (2024)
- [VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing](https://mlsanity.com/paper/2502.17258/) (2025)
- [KV-Edit: Training-Free Image Editing for Precise Background Preservation](https://mlsanity.com/paper/2502.17363/) (2025)
- [MultiBooth: Towards Generating All Your Concepts in an Image from Text](https://mlsanity.com/paper/2404.14239/) (2024)
- [EEdit: Rethinking the Spatial and Temporal Redundancy for Efficient Image Editing](https://mlsanity.com/paper/2503.10270/) (2025)
- [Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animation](https://mlsanity.com/paper/2509.16630/) (2025)
- [Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation](https://mlsanity.com/paper/2409.01055/) (2024)
- [Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices](https://mlsanity.com/paper/2410.11795/) (2024)
- [VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling](https://mlsanity.com/paper/2406.04321/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [InstantSwap: Fast Customized Concept Swapping across Sharp Shape Differences](https://mlsanity.com/paper/2412.01197/) (2024)
- [Follow-Your-Shape: Shape-Aware Image Editing via Trajectory-Guided Region Control](https://mlsanity.com/paper/2508.08134/) (2025)
- [ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichment](https://mlsanity.com/paper/2509.17818/) (2025)
- [Generative Video Motion Editing with 3D Point Tracks](https://mlsanity.com/paper/2512.02015/) (2025)
- [DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulation](https://mlsanity.com/paper/2503.16806/) (2025)
- [M-BEV: Masked BEV Perception for Robust Autonomous Driving](https://mlsanity.com/paper/2312.12144/) (2023)
- [Towards Multiple Character Image Animation Through Enhancing Implicit Decoupling](https://mlsanity.com/paper/2406.03035/) (2024)
- [ReFIR: Grounding Large Restoration Models with Retrieval Augmentation](https://mlsanity.com/paper/2410.05601/) (2024)
- [LLM-CAS: Dynamic Neuron Perturbation for Real-Time Hallucination Correction](https://mlsanity.com/paper/2512.18623/) (2025)
- [LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing](https://mlsanity.com/paper/2606.26740/) (2026)
- [VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models](https://mlsanity.com/paper/2403.05438/) (2024)

## Abstract (arXiv)

Text-based video editing has recently attracted considerable interest in changing the style or replacing the objects with a similar structure. Beyond this, we demonstrate that properties such as shape, size, location, motion, etc., can also be edited in videos. Our key insight is that the keyframe transformations of the specific internal feature (e.g., edge maps of objects or human pose), can easily propagate to other frames to provide generation guidance. We thus propose MagicStick, a controllable video editing method that edits the video properties by utilizing the transformation on the extracted internal control signals. In detail, to keep the appearance, we inflate both the pretrained image diffusion model and ControlNet to the temporal dimension and train low-rank adaptions (LORA) layers to fit the specific scenes. Then, in editing, we perform an inversion and editing framework. Differently, finetuned ControlNet is introduced in both inversion and generation for attention guidance with the proposed attention remix between the spatial attention maps of inversion and editing. Yet succinct, our method is the first method to show the ability of video property editing from the pre-trained text-to-image model. We present experiments on numerous examples within our unified framework. We also compare with shape-aware text-based editing and handcrafted motion video generation, demonstrating our superior temporal consistency and editing capability than previous works. The code and models are available on https://github.com/mayuelala/MagicStick.