# LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing

arXiv: https://arxiv.org/abs/2402.10294 · HTML: https://mlsanity.com/paper/2402.10294/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 35 of its 83 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning](https://mlsanity.com/paper/2205.00445/) (2022)
- [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://mlsanity.com/paper/2106.13353/) (2021)
- [Text-based Editing of Talking-head Video](https://mlsanity.com/paper/1906.01524/) (2019)
- [Evaluating the Factual Consistency of Large Language Models Through News Summarization](https://mlsanity.com/paper/2211.08412/) (2022)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI](https://mlsanity.com/paper/1909.02309/) (2019)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [Conceptual Metaphors Impact Perceptions of Human-AI Collaboration](https://mlsanity.com/paper/2008.02311/) (2020)
- [3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows](https://mlsanity.com/paper/2210.11603/) (2022)
- [StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement](https://mlsanity.com/paper/2202.06205/) (2022)
- [Opal: Multimodal Image Generation for News Illustration](https://mlsanity.com/paper/2204.09007/) (2022)
- [A Zero-Shot Language Agent for Computer Control with Structured Reflection](https://mlsanity.com/paper/2310.08740/) (2023)
- [AI Song Contest: Human-AI Co-Creation in Songwriting](https://mlsanity.com/paper/2010.05388/) (2020)
- [Rescribe: Authoring and Automatically Editing Audio Descriptions](https://mlsanity.com/paper/2010.03667/) (2020)
- [AVscript: Accessible Video Editing with Audio-Visual Scripts](https://mlsanity.com/paper/2302.14117/) (2023)

## Built on by (most cited first, 30 of 47)

- [UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing](https://mlsanity.com/paper/2402.13185/) (2024)
- [InstructX: Towards Unified Visual Editing with MLLM Guidance](https://mlsanity.com/paper/2510.08485/) (2025)
- [EditDuet: A Multi-Agent System for Video Non-Linear Editing](https://mlsanity.com/paper/2509.10761/) (2025)
- [UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist](https://mlsanity.com/paper/2511.08521/) (2025)
- [OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMs](https://mlsanity.com/paper/2405.03901/) (2024)
- [From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysis](https://mlsanity.com/paper/2502.11919/) (2025)
- [Predicting and Understanding Human Action Decisions: Insights from Large Language Models and Cognitive Instance-Based Learning](https://mlsanity.com/paper/2407.09281/) (2024)
- [Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learning](https://mlsanity.com/paper/2512.09924/) (2025)
- [MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation](https://mlsanity.com/paper/2407.12871/) (2024)
- [Lotus: Creating Short Videos From Long Videos With Abstractive and Extractive Summarization](https://mlsanity.com/paper/2502.07096/) (2025)
- [From Shots to Stories: LLM-Assisted Video Editing with Unified Language Representations](https://mlsanity.com/paper/2505.12237/) (2025)
- ["Before, I Asked My Mom, Now I Ask ChatGPT": Visual Privacy Management with Generative AI for Blind and Low-Vision People](https://mlsanity.com/paper/2507.00286/) (2025)
- [From Long Videos to Engaging Clips: A Human-Inspired Video Editing Framework with Multimodal Narrative Understanding](https://mlsanity.com/paper/2507.02790/) (2025)
- [Prompt-Driven Agentic Video Editing System: Autonomous Comprehension of Long-Form, Story-Driven Media](https://mlsanity.com/paper/2509.16811/) (2025)
- [A Benchmark and Multi-Agent System for Instruction-driven Cinematic Video Compilation](https://mlsanity.com/paper/2604.10456/) (2026)
- [Shot Sequence Ordering for Video Editing: Benchmarks, Metrics, and Cinematology-Inspired Computing Methods](https://mlsanity.com/paper/2503.17975/) (2025)
- [Learning to Highlight Audio by Watching Movies](https://mlsanity.com/paper/2505.12154/) (2025)
- [A Challenge to Build Neuro-Symbolic Video Agents](https://mlsanity.com/paper/2505.13851/) (2025)
- [LLM-Agent-Controller: A Universal Multi-Agent Large Language Model System as a Control Engineer](https://mlsanity.com/paper/2505.19567/) (2025)
- [LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants](https://mlsanity.com/paper/2507.05515/) (2025)
- [Reframe Anything: LLM Agent for Open World Video Reframing](https://mlsanity.com/paper/2403.06070/) (2024)
- [Survey of User Interface Design and Interaction Techniques in Generative AI Applications](https://mlsanity.com/paper/2410.22370/) (2024)
- [Navigating the Unknown: A Chat-Based Collaborative Interface for Personalized Exploratory Tasks](https://mlsanity.com/paper/2410.24032/) (2024)
- [Hookpad Aria: A Copilot for Songwriters](https://mlsanity.com/paper/2502.08122/) (2025)
- [Mobile-Agent-V: A Video-Guided Approach for Effortless and Efficient Operational Knowledge Injection in Mobile Automation](https://mlsanity.com/paper/2505.13887/) (2025)
- [Cooperative Design Optimization through Natural Language Interaction](https://mlsanity.com/paper/2508.16077/) (2025)
- [AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation](https://mlsanity.com/paper/2603.28366/) (2026)
- [Mobile-Agent-V: A Video-Guided Approach for Effortless and Efficient Operational Knowledge Injection in Mobile Automation](https://mlsanity.com/paper/2502.17110/) (2025)
- [MapStory: Prototyping Editable Map Animations with LLM Agents](https://mlsanity.com/paper/2505.21966/) (2025)
- [Multi-Branch Collaborative Learning Network for Video Quality Assessment in Industrial Video Search](https://mlsanity.com/paper/2502.05924/) (2025)

## Abstract (arXiv)

Video creation has become increasingly popular, yet the expertise and effort required for editing often pose barriers to beginners. In this paper, we explore the integration of large language models (LLMs) into the video editing workflow to reduce these barriers. Our design vision is embodied in LAVE, a novel system that provides LLM-powered agent assistance and language-augmented editing features. LAVE automatically generates language descriptions for the user's footage, serving as the foundation for enabling the LLM to process videos and assist in editing tasks. When the user provides editing objectives, the agent plans and executes relevant actions to fulfill them. Moreover, LAVE allows users to edit videos through either the agent or direct UI manipulation, providing flexibility and enabling manual refinement of agent actions. Our user study, which included eight participants ranging from novices to proficient editors, demonstrated LAVE's effectiveness. The results also shed light on user perceptions of the proposed LLM-assisted editing paradigm and its impact on users' creativity and sense of co-creation. Based on these findings, we propose design implications to inform the future development of agent-assisted content editing.