mlsanityData to 7 Oct 2026

A Joint Sequence Fusion Model for Video Question Answering and Retrieval

arXiv 1808.02559 ↗First posted 7 Aug 2018cs.CVcited by 208

Not on the map. Why?

Abstract (arXiv)

We present an approach named JSFusion (Joint Sequence Fusion) that can measure semantic similarity between any pairs of multimodal sequence data (e.g. a video clip and a language sentence). Our multimodal matching network consists of two key components. First, the Joint Semantic Tensor composes a dense pairwise representation of two sequence data into a 3D tensor. Then, the Convolutional Hierarchical Decoder computes their similarity score by discovering hidden hierarchical matches between the two sequence modalities. Both modules leverage hierarchical attention mechanisms that learn to promote well-matched representation patterns while prune out misaligned ones in a bottom-up manner. Although the JSFusion is a universal model to be applicable to any multimodal sequence data, this work focuses on video-language tasks including multimodal retrieval and video QA. We evaluate the JSFusion model in three retrieval and VQA tasks in LSMDC, for which our model achieves the best performance reported so far. We also perform multiple-choice and movie retrieval tasks for the MSR-VTT dataset, on which our approach outperforms many state-of-the-art methods.

Key facts. Cited by 208 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 62; 2023: 34; 2024: 31; 2025: 22; 2026: 11.

208 papers.

Sort:

By area:

  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  2. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  3. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  4. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  7. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  8. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  9. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  10. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  11. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  12. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
38 more
  1. 2022Egocentric Video-Language Pretrainingcited by 231
  2. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  3. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  4. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  5. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  6. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  7. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  8. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  9. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  10. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  11. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  12. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  13. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115
  14. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  15. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  16. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  17. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  18. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  19. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  20. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  21. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  22. 2021SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Eventscited by 84
  23. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  24. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  25. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  26. 2022Disentangled Representation Learning for Text-Video Retrievalcited by 77
  27. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  28. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  29. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  30. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  31. 2021QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queriescited by 72
  32. 2021CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representationscited by 72
  33. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  34. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  35. 2021Video as Conditional Graph Hierarchy for Multi-Granular Question Answeringcited by 68
  36. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  37. 2022Video Graph Transformer for Video Question Answeringcited by 63
  38. 2021Object-aware Video-language Pre-training for Retrievalcited by 57

50 most cited of 208 All 208 (JSON)