# Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning

arXiv: https://arxiv.org/abs/1811.02765 · HTML: https://mlsanity.com/paper/1811.02765/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2018.

## Built on by (most cited first, 6 of 6)

- [ADMoE: Anomaly Detection with Mixture-of-Experts from Noisy Labels](https://mlsanity.com/paper/2208.11290/) (2022)
- [The MSR-Video to Text Dataset with Clean Annotations](https://mlsanity.com/paper/2102.06448/) (2021)
- [Adversarial Mixture Of Experts with Category Hierarchy Soft Constraint](https://mlsanity.com/paper/2007.12349/) (2020)
- [Context-aware Mixture-of-Experts for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2208.07109/) (2022)
- [UnfoldML: Cost-Aware and Uncertainty-Based Dynamic 2D Prediction for Multi-Stage Classification](https://mlsanity.com/paper/2210.15056/) (2022)
- [Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning](https://mlsanity.com/paper/2608.11013/) (2026)

## Abstract (arXiv)

Although promising results have been achieved in video captioning, existing models are limited to the fixed inventory of activities in the training corpus, and do not generalize to open vocabulary scenarios. Here we introduce a novel task, zero-shot video captioning, that aims at describing out-of-domain videos of unseen activities. Videos of different activities usually require different captioning strategies in many aspects, i.e. word selection, semantic construction, and style expression etc, which poses a great challenge to depict novel activities without paired training data. But meanwhile, similar activities share some of those aspects in common. Therefore, We propose a principled Topic-Aware Mixture of Experts (TAMoE) model for zero-shot video captioning, which learns to compose different experts based on different topic embeddings, implicitly transferring the knowledge learned from seen activities to unseen ones. Besides, we leverage external topic-related text corpus to construct the topic embedding for each activity, which embodies the most relevant semantic vectors within the topic. Empirical results not only validate the effectiveness of our method in utilizing semantic knowledge for video captioning, but also show its strong generalization ability when describing novel activities.