mlsanityData to 7 Oct 2026

HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training

arXiv 2212.14546 ↗First posted 30 Dec 2022cs.CV cs.CL cs.MMcited by 72

Not on the map. Why?

Abstract (arXiv)

Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-language pre-training, thus not fully exploiting the unique characteristic of video, i.e., temporal. In this paper, we propose a Hierarchical Temporal-Aware video-language pre-training framework, HiTeA, with two novel pre-training tasks for modeling cross-modal alignment between moments and texts as well as the temporal relations of video-text pairs. Specifically, we propose a cross-modal moment exploration task to explore moments in videos, which results in detailed video moment representation. Besides, the inherent temporal relations are captured by aligning video-text pairs as a whole in different time resolutions with multi-modal temporal relation exploration task. Furthermore, we introduce the shuffling test to evaluate the temporal reliance of datasets and video-language pre-training models. We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e.g., SSv2-Template and SSv2-Label) with 8.6% and 11.1% improvement respectively. HiTeA also demonstrates strong generalization ability when directly transferred to downstream tasks in a zero-shot manner. Models and demo will be available on ModelScope.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 68 of its 80 references that are in the map. First posted on arXiv 30 Dec 2022.

Builds on

68 of 80 references.

References · 68

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
38 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2017Dense-Captioning Events in Videoscited by 789
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  10. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  11. 2017Localizing Moments in Video with Natural Languagecited by 660
  12. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  13. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  14. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  15. 2021ImageNet-21K Pretraining for the Massescited by 607
  16. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  17. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  18. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  19. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  20. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  21. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  22. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  23. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  24. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  25. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  26. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  27. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  28. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  29. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  30. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  31. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  32. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  33. 2015A Dataset for Movie Descriptioncited by 195
  34. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  35. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  36. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  37. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  38. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 27; 2024: 27; 2025: 12; 2026: 6.

72 papers.

Sort:

By area:

  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  5. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  6. 2023Video Understanding with Large Language Models: A Surveycited by 203
  7. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  8. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  9. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  10. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  11. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151
  12. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
38 more
  1. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  2. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  3. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  4. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  5. 2024Language Repository for Long Video Understandingcited by 55
  6. 2024Gramian Multimodal Representation Learning and Alignmentcited by 52
  7. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  8. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  9. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  10. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41
  11. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  12. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  13. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  14. 2023Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarkscited by 33
  15. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32
  16. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  17. 2023Text-Conditioned Resampler For Long Form Video Understandingcited by 24
  18. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 24
  19. 2023ViLA: Efficient Video-Language Alignment for Video Question Answeringcited by 23
  20. 2024End-to-End Video Question Answering with Frame Scoring Mechanisms and Adaptive Samplingcited by 19
  21. 2025Principled Multimodal Representation Learningcited by 19
  22. 2023VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blendingcited by 17
  23. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  24. 2025A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similaritycited by 15
  25. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  26. 2024Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answeringcited by 14
  27. 2023Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decouplingcited by 12
  28. 2023COSA: Concatenated Sample Pretrained Vision-Language Foundation Modelcited by 11
  29. 2024Enhancing Temporal Modeling of Video LLMs via Time Gatingcited by 10
  30. 2024Every Shot Counts: Using Exemplars for Repetition Counting in Videoscited by 8
  31. 2024EA-VTR: Event-Aware Video-Text Retrievalcited by 8
  32. 2024PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adaptercited by 7
  33. 2025EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real Worldcited by 7
  34. 2025RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graphcited by 7
  35. 2023Recursive Visual Programmingcited by 6
  36. 2024Towards Long Video Understanding via Fine-detailed Video Story Generationcited by 6
  37. 2025Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representationcited by 6
  38. 2024M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrievalcited by 5

50 most cited of 72 All 72 (JSON)