mlsanityData to 14 Oct 2026

Multi-sentence Video Grounding for Long Video Generation

arXiv 2407.13219 ↗First posted 18 Jul 2024cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Video generation has witnessed great success recently, but their application in generating long videos still remains challenging due to the difficulty in maintaining the temporal consistency of generated videos and the high memory cost during generation. To tackle the problems, in this paper, we propose a brave and new idea of Multi-sentence Video Grounding for Long Video Generation, connecting the massive video moment retrieval to the video generation task for the first time, providing a new paradigm for long video generation. The method of our work can be summarized as three steps: (i) We design sequential scene text prompts as the queries for video grounding, utilizing the massive video moment retrieval to search for video moment segments that meet the text requirements in the video database. (ii) Based on the source frames of retrieved video moment segments, we adopt video editing methods to create new video content while preserving the temporal consistency of the retrieved video. Since the editing can be conducted segment by segment, and even frame by frame, it largely reduces the memory cost. (iii) We also attempt video morphing and personalized generation methods to improve the subject consistency of long video generation, providing ablation experimental results for the subtasks of long video generation. Our approach seamlessly extends the development in image/video editing, video morphing and personalized generation, and video grounding to the long video generation, offering effective solutions for generating long videos at low memory cost.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 38 of its 60 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

38 of 60 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  12. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
26 more
  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  4. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  5. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  6. 2023ModelScope Text-to-Video Technical Reportcited by 658
  7. 2017TALL: Temporal Activity Localization via Language Querycited by 584
  8. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  9. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  10. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  11. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  12. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 255
  13. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  14. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 180
  15. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 171
  16. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 167
  17. 2023VidToMe: Video Token Merging for Zero-Shot Video Editingcited by 113
  18. 2021Interventional Video Grounding with Dual Contrastive Learningcited by 94
  19. 2021Negative Sample Matters: A Renaissance of Metric Learning for Temporal Groundingcited by 90
  20. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 90
  21. 2023DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generationcited by 75
  22. 2023DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphingcited by 67
  23. 2021A Survey on Temporal Sentence Grounding in Videoscited by 29
  24. 2023LLM4VG: Large Language Models Evaluation for Video Groundingcited by 8
  25. 2022Modal-specific Pseudo Query Generation for Video Corpus Moment Retrievalcited by 6
  26. 2023LOVECon: Text-driven Training-Free Long Video Editing with ControlNetcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Modularized Dynamic-Granularity Video LLM for Multi-Event Long Video Understandingcited by 0