mlsanityData to 7 Oct 2026

Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video Localization

arXiv 2401.08232 ↗First posted 16 Jan 2024cs.CVcited by 0

On the map: Diffusion models → Video generation

Abstract (arXiv)

Natural Language Video Localization (NLVL), grounding phrases from natural language descriptions to corresponding video segments, is a complex yet critical task in video understanding. Despite ongoing advancements, many existing solutions lack the capability to globally capture temporal dynamics of the video data. In this study, we present a novel approach to NLVL that aims to address this issue. Our method involves the direct generation of a global 2D temporal map via a conditional denoising diffusion process, based on the input video and language query. The main challenges are the inherent sparsity and discontinuity of a 2D temporal map in devising the diffusion decoder. To address these challenges, we introduce a multi-scale technique and develop an innovative diffusion decoder. Our approach effectively encapsulates the interaction between the query and video data across various time scales. Experiments on the Charades and DiDeMo datasets underscore the potency of our design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 36 of its 44 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

36 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model31 Aug

Other references · 35

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  6. 2017Localizing Moments in Video with Natural Languagecited by 660
  7. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  8. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  9. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  10. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  11. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  12. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
23 more
  1. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 283
  2. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 279
  3. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 234
  4. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  5. 2020Local-Global Video-Text Interactions for Temporal Groundingcited by 188
  6. 2018To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regressioncited by 157
  7. 2018MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustmentcited by 145
  8. 2019Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videoscited by 135
  9. 2018Multilevel Language and Vision Integration for Text-to-Clip Retrievalcited by 128
  10. 2021Interventional Video Grounding with Dual Contrastive Learningcited by 94
  11. 2021Boundary Proposal Network for Two-Stage Natural Language Video Localizationcited by 88
  12. 2019ExCL: Extractive Clip Localization Using Natural Language Descriptionscited by 82
  13. 2019Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videoscited by 79
  14. 2020Jointly Cross- and Self-Modal Graph Attention Network for Query-Based Moment Localizationcited by 74
  15. 2023Diffusion Action Segmentationcited by 74
  16. 2022Latent Diffusion Energy-Based Model for Interpretable Text Modelingcited by 67
  17. 2023MomentDiff: Generative Video Moment Retrieval from Random to Realcited by 58
  18. 2021Natural Language Video Localization: A Revisit in Span-based Question Answering Frameworkcited by 55
  19. 2021On Pursuit of Designing Multi-modal Transformer for Video Groundingcited by 54
  20. 2020VLG-Net: Video-Language Graph Matching Network for Video Groundingcited by 50
  21. 2020Multi-Scale 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 29
  22. 2021Natural Language Video Localization with Learnable Moment Proposalscited by 27
  23. 2020Frame-wise Cross-modal Matching for Video Moment Retrievalcited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Commonsense for Zero-Shot Natural Language Video Localization
  2. 2026Masked Diffusion Vision-Language Models for Temporal Action Localization
  3. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability
  4. 2025SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capability
  5. 2022Explore-And-Match: Bridging Proposal-Based and Proposal-Free With Transformer for Sentence Grounding in Videos
  6. 2024Training-free Video Temporal Grounding using Large-scale Pre-trained Models
  7. 2026Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformers
  8. 2023DiffusionVMR: Diffusion Model for Joint Video Moment Retrieval and Highlight Detection