# LoRA-Edit: Controllable First-Frame-Guided Video Editing via Mask-Aware LoRA Fine-Tuning

arXiv: https://arxiv.org/abs/2506.10082 · HTML: https://mlsanity.com/paper/2506.10082/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 30 of its 35 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Video-P2P: Video Editing with Cross-attention Control](https://mlsanity.com/paper/2303.04761/) (2023)
- [Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation](https://mlsanity.com/paper/2306.07954/) (2023)
- [FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing](https://mlsanity.com/paper/2310.05922/) (2023)
- [AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks](https://mlsanity.com/paper/2403.14468/) (2024)
- [Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution](https://mlsanity.com/paper/2501.11561/) (2025)
- [Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models](https://mlsanity.com/paper/2303.17599/) (2023)
- [Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise](https://mlsanity.com/paper/2501.08331/) (2025)
- [VidToMe: Video Token Merging for Zero-Shot Video Editing](https://mlsanity.com/paper/2312.10656/) (2023)
- [ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling](https://mlsanity.com/paper/2501.02487/) (2025)
- [HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation](https://mlsanity.com/paper/2505.04512/) (2025)
- [I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models](https://mlsanity.com/paper/2405.16537/) (2024)
- [DragVideo: Interactive Drag-style Video Editing](https://mlsanity.com/paper/2312.02216/) (2023)
- [Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis](https://mlsanity.com/paper/2312.13834/) (2023)
- [Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models](https://mlsanity.com/paper/2409.07452/) (2024)
- [SAVE: Protagonist Diversification with Structure Agnostic Video Editing](https://mlsanity.com/paper/2312.02503/) (2023)
- [MTV-Inpaint: Multi-Task Long Video Inpainting](https://mlsanity.com/paper/2503.11412/) (2025)

## Built on by (most cited first, 28 of 28)

- [In-Context Learning with Unpaired Clips for Instruction-based Video Editing](https://mlsanity.com/paper/2510.14648/) (2025)
- [Generative Video Motion Editing with 3D Point Tracks](https://mlsanity.com/paper/2512.02015/) (2025)
- [PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion](https://mlsanity.com/paper/2606.01399/) (2026)
- [VideoCanvas: Unified Video Completion from Arbitrary Spatiotemporal Patches via In-Context Conditioning](https://mlsanity.com/paper/2510.08555/) (2025)
- [EditCast3D: Single-Frame-Guided 3D Editing with Video Propagation and View Selection](https://mlsanity.com/paper/2510.13652/) (2025)
- [Continuous Control of Editing Models via Adaptive-Origin Guidance](https://mlsanity.com/paper/2602.03826/) (2026)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [AVControl: Efficient Framework for Training Audio-Visual Controls](https://mlsanity.com/paper/2603.24793/) (2026)
- [IC-Effect: Precise and Efficient Video Effects Editing via In-Context Learning](https://mlsanity.com/paper/2512.15635/) (2025)
- [NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editing](https://mlsanity.com/paper/2603.02802/) (2026)
- [Versatile Editing of Video Content, Actions, and Dynamics without Training](https://mlsanity.com/paper/2603.17989/) (2026)
- [VideoGuard: Protecting Video Content from Unauthorized Editing](https://mlsanity.com/paper/2508.03480/) (2025)
- [Over++: Generative Video Compositing for Layer Interaction Effects](https://mlsanity.com/paper/2512.19661/) (2025)
- [Let Your Image Move with Your Motion! -- Implicit Multi-Object Multi-Motion Transfer](https://mlsanity.com/paper/2603.01000/) (2026)
- [RewardFlow: Generate Images by Optimizing What You Reward](https://mlsanity.com/paper/2604.08536/) (2026)
- [Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Framework](https://mlsanity.com/paper/2605.23891/) (2026)
- [ElasticTTT: Prior-Preserving Test-Time Tuning for Video Editing](https://mlsanity.com/paper/2607.21529/) (2026)
- [Compression as Adaptation: Implicit Visual Representation with Diffusion Foundation Models](https://mlsanity.com/paper/2603.07615/) (2026)
- [Composing Driving Worlds through Disentangled Control for Adversarial Scenario Generation](https://mlsanity.com/paper/2603.12864/) (2026)
- [ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer](https://mlsanity.com/paper/2603.15478/) (2026)
- [From Gallery to Wrist: Realistic 3D Bracelet Insertion in Videos](https://mlsanity.com/paper/2507.20331/) (2025)
- [ReLumix: Extending Image Relighting to Video via Video Diffusion Models](https://mlsanity.com/paper/2509.23769/) (2025)
- [Learning Stochastic Bridges for Video Object Removal via Video-to-Video Translation](https://mlsanity.com/paper/2601.12066/) (2026)
- [Controllable Video Object Insertion via Multi-View Priors](https://mlsanity.com/paper/2604.14556/) (2026)
- [RefDecoder: Enhancing Visual Generation with Conditional Video Decoding](https://mlsanity.com/paper/2605.15196/) (2026)
- [Sound Sparks Motion: Audio and Text Tuning for Video Editing](https://mlsanity.com/paper/2605.15307/) (2026)
- [DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation](https://mlsanity.com/paper/2606.31918/) (2026)
- [Video Generation Models: A Survey of Post-Training and Alignment](https://mlsanity.com/paper/2610.00812/) (2026)

## Abstract (arXiv)

Video editing using diffusion models has achieved remarkable results in generating high-quality edits for videos. However, current methods often rely on large-scale pretraining, limiting flexibility for specific edits. First-frame-guided editing provides control over the first frame, but lacks fine-grained control over the edit's subsequent temporal evolution. To address this, we propose a mask-based LoRA (Low-Rank Adaptation) tuning method that adapts pretrained Image-to-Video models for flexible video editing. Our key innovation is using a spatiotemporal mask to strategically guide the LoRA fine-tuning process. This teaches the model two distinct skills: first, to interpret the mask as a command to either preserve content from the source video or generate new content in designated regions. Second, for these generated regions, LoRA learns to synthesize either temporally consistent motion inherited from the video or novel appearances guided by user-provided reference frames. This dual-capability LoRA grants users control over the edit's entire temporal evolution, allowing complex transformations like an object rotating or a flower blooming. Experimental results show our method achieves superior video editing performance compared to baseline methods. The code and video results are available at our project website: https://cjeen.github.io/LoRAEdit.