mlsanityData to 7 Oct 2026

UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection

arXiv 2203.12745 ↗First posted 23 Mar 2022cs.CVcited by 148

Not on the map. Why?

Abstract (arXiv)

Finding relevant moments and highlights in videos according to natural language queries is a natural and highly valuable common need in the current video content explosion era. Nevertheless, jointly conducting moment retrieval and highlight detection is an emerging research topic, even though its component problems and some related tasks have already been studied for a while. In this paper, we present the first unified framework, named Unified Multi-modal Transformers (UMT), capable of realizing such joint optimization while can also be easily degenerated for solving individual problems. As far as we are aware, this is the first scheme to integrate multi-modal (visual-audio) learning for either joint optimization or the individual moment retrieval task, and tackles moment retrieval as a keypoint detection problem using a novel query generator and query decoder. Extensive comparisons with existing methods and ablation studies on QVHighlights, Charades-STA, YouTube Highlights, and TVSum datasets demonstrate the effectiveness, superiority, and flexibility of the proposed method under various settings. Source code and pre-trained models are available at https://github.com/TencentARC/UMT.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 23 Mar 2022.

Builds on

30 of 52 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2016Layer Normalizationcited by 5,579
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2019Objects as Pointscited by 1,172
  10. 2018CornerNet: Detecting Objects as Paired Keypointscited by 763
  11. 2017Focal Loss for Dense Object Detectioncited by 749
  12. 2017Localizing Moments in Video with Natural Languagecited by 660
18 more
  1. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  2. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  3. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  4. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 283
  5. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  6. 2018MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustmentcited by 145
  7. 2016Video Summarization with Long Short-term Memorycited by 142
  8. 2021QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queriescited by 72
  9. 2019Less is More: Learning Highlight Detection from Video Durationcited by 60
  10. 2020MINI-Net: Multiple Instance Ranking Network for Video Highlight Detectioncited by 42
  11. 2016Video2GIF: Automatic Generation of Animated GIFs from Videocited by 41
  12. 2021Cross-category Video Highlight Detection via Set-based Learningcited by 38
  13. 2020Learning Trailer Moments in Full-Length Moviescited by 37
  14. 2016To Click or Not To Click: Automatic Selection of Beautiful Thumbnails from Videoscited by 33
  15. 2015Unsupervised Extraction of Video Highlights Via Robust Recurrent Auto-encoderscited by 30
  16. 2016Learning Joint Representations of Videos and Sentences with Web Image Searchcited by 17
  17. 2019Sentence Specified Dynamic Video Thumbnail Generationcited by 8
  18. 2020Sentence Guided Temporal Modulation for Dynamic Video Thumbnail Generationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 25; 2024: 47; 2025: 47; 2026: 25.

148 papers.

Sort:

By area:

  1. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  2. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  3. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  4. 2023Query-Dependent Video Representation for Moment Retrieval and Highlight Detectioncited by 166
  5. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116
  6. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  7. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  8. 2023Knowing Where to Focus: Event-aware Transformer for Video Groundingcited by 79
  9. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  10. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66
  11. 2024TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detectioncited by 59
  12. 2023MomentDiff: Generative Video Moment Retrieval from Random to Realcited by 58
38 more
  1. 2024Gramian Multimodal Representation Learning and Alignmentcited by 52
  2. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 49
  3. 2024StyleMaster: Stylize Your Video with Artistic Generation and Translationcited by 47
  4. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  5. 2023MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformercited by 40
  6. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  7. 2024R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Groundingcited by 34
  8. 2023Correlation-Guided Query-Dependency Calibration for Video Temporal Groundingcited by 31
  9. 2024Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrievalcited by 30
  10. 2023BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videoscited by 29
  11. 2023Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrievalcited by 27
  12. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 26
  13. 2024SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Groundingcited by 25
  14. 2024Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detectioncited by 25
  15. 2024FlashVTG: Feature Layering and Adaptive Score Handling Network for Video Temporal Groundingcited by 23
  16. 2024Saliency-Guided DETR for Moment Retrieval and Highlight Detectioncited by 21
  17. 2024Multimodal Action Quality Assessmentcited by 20
  18. 2024VTG-GPT: Tuning-Free Zero-Shot Video Temporal Grounding with GPTcited by 19
  19. 2025Principled Multimodal Representation Learningcited by 19
  20. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  21. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videoscited by 18
  22. 2024Chrono: A Simple Blueprint for Representing Time in MLLMscited by 17
  23. 2024A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trendscited by 16
  24. 2024UniMD: Towards Unifying Moment Retrieval and Temporal Action Detectioncited by 16
  25. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  26. 2023RGNet: A Unified Clip Retrieval and Grounding Network for Long Videoscited by 15
  27. 2025A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similaritycited by 15
  28. 2023Grounding-Prompter: Prompting LLM with Multimodal Information for Temporal Sentence Grounding in Long Videoscited by 14
  29. 2024Do Language Models Understand Time?cited by 14
  30. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  31. 2023SCANet: Scene Complexity Aware Network for Weakly-Supervised Video Moment Retrievalcited by 13
  32. 2024Unified Static and Dynamic Network: Efficient Temporal Filtering for Video Groundingcited by 12
  33. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  34. 2023DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Groundingcited by 11
  35. 2023Boundary-Denoising for Video Activity Localizationcited by 10
  36. 2024Beyond Uncertainty: Evidential Deep Learning for Robust Video Temporal Groundingcited by 10
  37. 2024Context-Enhanced Video Moment Retrieval with Large Language Modelscited by 9
  38. 2025Moment of Untruth: Dealing with Negative Queries in Video Moment Retrievalcited by 9

50 most cited of 148 All 148 (JSON)