mlsanityData to 13 Oct 2026

Cross-Modal and Hierarchical Modeling of Video and Text

arXiv 1810.07212 ↗First posted 16 Oct 2018cs.CVcited by 103

Not on the map. Why?

Abstract (arXiv)

Visual data and text data are composed of information at multiple granularities. A video can describe a complex scene that is composed of multiple clips or shots, where each depicts a semantically coherent event or action. Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story. In this paper, we investigate the modeling techniques for such hierarchical sequential data where there are correspondences across multiple modalities. Specifically, we introduce hierarchical sequence embedding (HSE), a generic model for embedding sequential data of different modalities into hierarchically semantic spaces, with either explicit or implicit correspondence information. We perform empirical studies on large-scale video and paragraph retrieval datasets and demonstrated superior performance by the proposed methods. Furthermore, we examine the effectiveness of our learned embeddings when applied to downstream tasks. We show its utility in zero-shot action recognition and video captioning.

Key facts. Cited by 103 papers in the map (data through 13 Oct 2026). First posted on arXiv 16 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 30; 2022: 30; 2023: 18; 2024: 17; 2025: 6; 2026: 2.

103 papers.

Sort:

By area:

  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,176
  2. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 650
  3. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 600
  4. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  5. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 318
  6. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  7. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 245
  8. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  9. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  10. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  11. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  12. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
38 more
  1. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  2. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  3. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  4. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 103
  5. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  6. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  7. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  8. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  9. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  10. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  11. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  12. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  13. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  14. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  15. 2021Video Corpus Moment Retrieval with Contrastive Learningcited by 60
  16. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  17. 2021Object-aware Video-language Pre-training for Retrievalcited by 57
  18. 2021Motion-aware Contrastive Video Representation Learning via Foreground-background Mergingcited by 45
  19. 2023Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learningcited by 45
  20. 2023VideoXum: Cross-modal Visual and Textural Summarization of Videoscited by 41
  21. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 40
  22. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
  23. 2022Hierarchical Local-Global Transformer for Temporal Sentence Groundingcited by 38
  24. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 38
  25. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32
  26. 2021HANet: Hierarchical Alignment Networks for Video-Text Retrievalcited by 31
  27. 2022Visual Abductive Reasoningcited by 30
  28. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  29. 2023Text with Knowledge Graph Augmented Transformer for Video Captioningcited by 28
  30. 2021Masking Modalities for Cross-modal Video Retrievalcited by 23
  31. 2023Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?cited by 21
  32. 2023Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignmentcited by 21
  33. 2021Video-Text Pre-training with Learned Regionscited by 20
  34. 2022Text-Adaptive Multiple Visual Prototype Matching for Video-Text Retrievalcited by 20
  35. 2023Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Bankscited by 20
  36. 2023Tagging before Alignment: Integrating Multi-Modal Tags for Video-Text Retrievalcited by 18
  37. 2022TempCLR: Temporal Alignment Representation with Contrastive Learningcited by 15
  38. 2023Deep Learning for Video-Text Retrieval: a Reviewcited by 15

50 most cited of 103 All 103 (JSON)