# REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editing

arXiv: https://arxiv.org/abs/2505.18880 · HTML: https://mlsanity.com/paper/2505.18880/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 24 May 2025.

## Builds on (most cited first, 28 of 28)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://mlsanity.com/paper/2401.11817/) (2024)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models](https://mlsanity.com/paper/2212.08037/) (2022)
- [One-Minute Video Generation with Test-Time Training](https://mlsanity.com/paper/2504.05298/) (2025)
- [CLIP-It! Language-Guided Video Summarization](https://mlsanity.com/paper/2107.00650/) (2021)
- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [VideoXum: Cross-modal Visual and Textural Summarization of Videos](https://mlsanity.com/paper/2303.12060/) (2023)
- [Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications](https://mlsanity.com/paper/2311.05876/) (2023)
- [Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models](https://mlsanity.com/paper/2402.08670/) (2024)
- [Scaling Up Video Summarization Pretraining with Large Language Models](https://mlsanity.com/paper/2404.03398/) (2024)
- [Learning Fine-Grained Grounded Citations for Attributed Large Language Models](https://mlsanity.com/paper/2408.04568/) (2024)
- [Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides](https://mlsanity.com/paper/2208.08080/) (2022)
- [Towards Automated Movie Trailer Generation](https://mlsanity.com/paper/2404.03477/) (2024)
- [ScreenWriter: Automatic Screenplay Generation and Movie Summarisation](https://mlsanity.com/paper/2410.19809/) (2024)
- [Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversations](https://mlsanity.com/paper/2406.15000/) (2024)
- [TeaserGen: Generating Teasers for Long Documentaries](https://mlsanity.com/paper/2410.05586/) (2024)
- ["Previously on ..." From Recaps to Story Summarization](https://mlsanity.com/paper/2405.11487/) (2024)

## Built on by (most cited first, 1 of 1)

- [Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset](https://mlsanity.com/paper/2510.15742/) (2025)

## Abstract (arXiv)

Short videos are an effective tool for promoting contents and improving knowledge accessibility. While existing extractive video summarization methods struggle to produce a coherent narrative, existing abstractive methods cannot `quote' from the input videos, i.e., inserting short video clips in their outputs. In this work, we explore novel video editing models for generating shorts that feature a coherent narrative with embedded video insertions extracted from a long input video. We propose a novel retrieval-embedded generation framework that allows a large language model to quote multimodal resources while maintaining a coherent narrative. Our proposed REGen system first generates the output story script with quote placeholders using a finetuned large language model, and then uses a novel retrieval model to replace the quote placeholders by selecting a video clip that best supports the narrative from a pool of candidate quotable video clips. We examine the proposed method on the task of documentary teaser generation, where short interview insertions are commonly used to support the narrative of a documentary. Our objective evaluations show that the proposed method can effectively insert short video clips while maintaining a coherent narrative. In a subjective survey, we show that our proposed method outperforms existing abstractive and extractive approaches in terms of coherence, alignment, and realism in teaser generation.