# ExpressEdit: Video Editing with Natural Language and Sketching arXiv: https://arxiv.org/abs/2403.17693 ยท HTML: https://mlsanity.com/paper/2403.17693/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 19 of its 118 references that are in the map. - First posted on arXiv 26 Mar 2024. ## Builds on (most cited first, 19 of 19) - [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022) - [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016) - [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023) - [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022) - [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021) - [StableVideo: Text-driven Consistency-aware Diffusion Video Editing](https://mlsanity.com/paper/2308.09592/) (2023) - [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023) - [To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making](https://mlsanity.com/paper/2102.09692/) (2021) - [Text-based Editing of Talking-head Video](https://mlsanity.com/paper/1906.01524/) (2019) - [Iterative Text-based Editing of Talking-heads Using Neural Retargeting](https://mlsanity.com/paper/2011.10688/) (2020) - [AVscript: Accessible Video Editing with Audio-Visual Scripts](https://mlsanity.com/paper/2302.14117/) (2023) - [B-Script: Transcript-based B-roll Video Editing with Recommendations](https://mlsanity.com/paper/1902.11216/) (2019) - [Soundini: Sound-Guided Diffusion for Natural Video Editing](https://mlsanity.com/paper/2304.06818/) (2023) - [Style-A-Video: Agile Diffusion for Arbitrary Text-based Video Style Transfer](https://mlsanity.com/paper/2305.05464/) (2023) - [RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling](https://mlsanity.com/paper/2208.06350/) (2022) - [HelpViz: Automatic Generation of Contextual Visual MobileTutorials from Text-Based Instructions](https://mlsanity.com/paper/2108.03356/) (2021) - [VideoMap: Supporting Video Editing Exploration, Brainstorming, and Prototyping in the Latent Space](https://mlsanity.com/paper/2211.12492/) (2022) - [Automated Conversion of Music Videos into Lyric Videos](https://mlsanity.com/paper/2308.14922/) (2023) - [Puppet Dubbing](https://mlsanity.com/paper/1902.04285/) (2019) ## Built on by (most cited first, 14 of 14) - [EditDuet: A Multi-Agent System for Video Non-Linear Editing](https://mlsanity.com/paper/2509.10761/) (2025) - [Lotus: Creating Short Videos From Long Videos With Abstractive and Extractive Summarization](https://mlsanity.com/paper/2502.07096/) (2025) - [Prompt-Driven Agentic Video Editing System: Autonomous Comprehension of Long-Form, Story-Driven Media](https://mlsanity.com/paper/2509.16811/) (2025) - [Shot Sequence Ordering for Video Editing: Benchmarks, Metrics, and Cinematology-Inspired Computing Methods](https://mlsanity.com/paper/2503.17975/) (2025) - [LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localization](https://mlsanity.com/paper/2512.02933/) (2025) - [AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation](https://mlsanity.com/paper/2603.28366/) (2026) - [MapStory: Prototyping Editable Map Animations with LLM Agents](https://mlsanity.com/paper/2505.21966/) (2025) - [Exploration vs. Fixation: Scaffolding Divergent and Convergent Thinking for Human-AI Co-Creation with Generative Models](https://mlsanity.com/paper/2512.18388/) (2025) - [Vidmento: Creating Video Stories Through Context-Aware Expansion With Generative Video](https://mlsanity.com/paper/2601.22013/) (2026) - [Zero-Shot Personalized Camera Motion Control for Image-to-Video Synthesis](https://mlsanity.com/paper/2504.09472/) (2025) - [Robots Influencing Humans to Reveal their Goals during Collaboration and Competition](https://mlsanity.com/paper/2609.05519/) (2026) - [Script-to-Slide Grounding: Grounding Script Sentences to Slide Objects for Automatic Instructional Video Generation](https://mlsanity.com/paper/2603.16931/) (2026) - [FeVOS: Foresight Expression Video Object Segmentation](https://mlsanity.com/paper/2606.25585/) (2026) - [Autoregressive Modeling of Film with Applications in Video Montage](https://mlsanity.com/paper/2607.14645/) (2026) ## Abstract (arXiv) Informational videos serve as a crucial source for explaining conceptual and procedural knowledge to novices and experts alike. When producing informational videos, editors edit videos by overlaying text/images or trimming footage to enhance the video quality and make it more engaging. However, video editing can be difficult and time-consuming, especially for novice video editors who often struggle with expressing and implementing their editing ideas. To address this challenge, we first explored how multimodality-natural language (NL) and sketching, which are natural modalities humans use for expression-can be utilized to support video editors in expressing video editing ideas. We gathered 176 multimodal expressions of editing commands from 10 video editors, which revealed the patterns of use of NL and sketching in describing edit intents. Based on the findings, we present ExpressEdit, a system that enables editing videos via NL text and sketching on the video frame. Powered by LLM and vision models, the system interprets (1) temporal, (2) spatial, and (3) operational references in an NL command and spatial references from sketching. The system implements the interpreted edits, which then the user can iterate on. An observational study (N=10) showed that ExpressEdit enhanced the ability of novice video editors to express and implement their edit ideas. The system allowed participants to perform edits more efficiently and generate more ideas by generating edits based on user's multimodal edit commands and supporting iterations on the editing commands. This work offers insights into the design of future multimodal interfaces and AI-based pipelines for video editing.