mlsanityData to 7 Oct 2026

HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

arXiv 2005.00200 ↗First posted 1 May 2020cs.CV cs.CL cs.LGcited by 362

Not on the map. Why?

Cited by 41 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions. Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains. We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities.

Key facts. Cited by 362 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 61; 2022: 87; 2023: 99; 2024: 64; 2025: 34; 2026: 17.

362 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  6. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  9. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  11. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  12. 2022Multimodal Learning with Transformers: A Surveycited by 399
38 more
  1. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  2. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  3. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  4. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  5. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  6. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  7. 2023Video Understanding with Large Language Models: A Surveycited by 203
  8. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  9. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  10. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  11. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  12. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  13. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  14. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  15. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  16. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  17. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  18. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  19. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  20. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 135
  21. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  22. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  23. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  24. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  25. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115
  26. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  27. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
  28. 2021Context-aware Biaffine Localizing Network for Temporal Sentence Groundingcited by 107
  29. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  30. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  31. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  32. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  33. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  34. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  35. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  36. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  37. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  38. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87

50 most cited of 362 All 362 (JSON)