# RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrieval

arXiv: https://arxiv.org/abs/2206.12845 · HTML: https://mlsanity.com/paper/2206.12845/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 26 Jun 2022.

## Placed in

- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [ActBERT: Learning Global-Local Video-Text Representations](https://mlsanity.com/paper/2011.07231/) (2020)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [AVLnet: Learning Audio-Visual Language Representations from Instructional Videos](https://mlsanity.com/paper/2006.09199/) (2020)
- [Learning a Text-Video Embedding from Incomplete and Heterogeneous Data](https://mlsanity.com/paper/1804.02516/) (2018)
- [Cross-Modal and Hierarchical Modeling of Video and Text](https://mlsanity.com/paper/1810.07212/) (2018)
- [TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment](https://mlsanity.com/paper/2108.09980/) (2021)
- [Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings](https://mlsanity.com/paper/1908.03477/) (2019)
- [COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning](https://mlsanity.com/paper/2011.00597/) (2020)
- [Dual Encoding for Zero-Example Video Retrieval](https://mlsanity.com/paper/1809.06181/) (2018)
- [Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval](https://mlsanity.com/paper/1906.04402/) (2019)
- [TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval](https://mlsanity.com/paper/2104.08271/) (2021)

## Built on by (most cited first, 11 of 11)

- [MoE-LLaVA: Mixture of Experts for Large Vision-Language Models](https://mlsanity.com/paper/2401.15947/) (2024)
- [Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models](https://mlsanity.com/paper/2407.14093/) (2024)
- [Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis](https://mlsanity.com/paper/2208.03051/) (2022)
- [Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model](https://mlsanity.com/paper/2406.19905/) (2024)
- [Exploiting Semantic Role Contextualized Video Features for Multi-Instance Text-Video Retrieval EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022](https://mlsanity.com/paper/2206.14381/) (2022)
- [An Overview of Challenges in Egocentric Text-Video Retrieval](https://mlsanity.com/paper/2306.04345/) (2023)
- [Reinforcement Learning-based Mixture of Vision Transformers for Video Violence Recognition](https://mlsanity.com/paper/2310.03108/) (2023)
- [Leveraging Modality Tags for Enhanced Cross-Modal Video Retrieval](https://mlsanity.com/paper/2504.01591/) (2025)
- [Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Intervention](https://mlsanity.com/paper/2309.09311/) (2023)
- [Repeating Words for Video-Language Retrieval with Coarse-to-Fine Objectives](https://mlsanity.com/paper/2508.14812/) (2025)
- [UNIFORM: Unifying Knowledge from Large-scale and Diverse Pre-trained Models](https://mlsanity.com/paper/2508.19498/) (2025)

## Abstract (arXiv)

Seas of videos are uploaded daily with the popularity of social channels; thus, retrieving the most related video contents with user textual queries plays a more crucial role. Most methods consider only one joint embedding space between global visual and textual features without considering the local structures of each modality. Some other approaches consider multiple embedding spaces consisting of global and local features separately, ignoring rich inter-modality correlations. We propose a novel mixture-of-expert transformer RoME that disentangles the text and the video into three levels; the roles of spatial contexts, temporal contexts, and object contexts. We utilize a transformer-based attention mechanism to fully exploit visual and text embeddings at both global and local levels with mixture-of-experts for considering inter-modalities and structures' correlations. The results indicate that our method outperforms the state-of-the-art methods on the YouCook2 and MSR-VTT datasets, given the same visual backbone without pre-training. Finally, we conducted extensive ablation studies to elucidate our design choices.