mlsanityData to 14 Oct 2026

GEM-VPC: A dual Graph-Enhanced Multimodal integration for Video Paragraph Captioning

arXiv 2410.09377 ↗First posted 12 Oct 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Video Paragraph Captioning (VPC) aims to generate paragraph captions that summarises key events within a video. Despite recent advancements, challenges persist, notably in effectively utilising multimodal signals inherent in videos and addressing the long-tail distribution of words. The paper introduces a novel multimodal integrated caption generation framework for VPC that leverages information from various modalities and external knowledge bases. Our framework constructs two graphs: a 'video-specific' temporal graph capturing major events and interactions between multimodal information and commonsense knowledge, and a 'theme graph' representing correlations between words of a specific theme. These graphs serve as input for a transformer network with a shared encoder-decoder architecture. We also introduce a node selection module to enhance decoding efficiency by selecting the most relevant nodes from the graphs. Our results demonstrate superior performance across benchmark datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 64 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

47 of 64 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  10. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  12. 2021How Attentive are Graph Attention Networks?cited by 835
35 more
  1. 2017Dense-Captioning Events in Videoscited by 793
  2. 2021AST: Audio Spectrogram Transformercited by 643
  3. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  4. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  5. 2018Graph Convolutional Networks for Text Classificationcited by 345
  6. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  7. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  8. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  9. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  10. 2019Graph Convolutional Networks for Temporal Action Localizationcited by 208
  11. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  12. 2020Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentencescited by 118
  13. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 116
  14. 2020Multi-modal Dense Video Captioningcited by 113
  15. 2020Spatio-Temporal Graph for Video Captioning with Knowledge Distillationcited by 109
  16. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  17. 2017Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Imagescited by 101
  18. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  19. 2020MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioningcited by 79
  20. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  21. 2018Grounded Video Descriptioncited by 78
  22. 2020A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformercited by 76
  23. 2019Watch, Listen and Tell: Multi-modal Weakly Supervised Dense Event Captioningcited by 38
  24. 2021Unified Graph Structured Models for Video Understandingcited by 38
  25. 2018Move Forward and Tell: A Progressive Generator of Video Descriptionscited by 35
  26. 2018Adversarial Inference for Multi-Sentence Video Descriptioncited by 31
  27. 2022Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detectioncited by 30
  28. 2023Text with Knowledge Graph Augmented Transformer for Video Captioningcited by 28
  29. 2022VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioningcited by 25
  30. 2021Towards Diverse Paragraph Captioning for Untrimmed Videoscited by 22
  31. 2022VLCap: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioningcited by 19
  32. 2019A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captionscited by 15
  33. 2019Improving Image Captioning by Leveraging Knowledge Graphscited by 11
  34. 2021AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generationcited by 11
  35. 2024SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytellingcited by 5

Built on by

Not cited yet.