# Multi-Modal Video Topic Segmentation with Dual-Contrastive Domain Adaptation

arXiv: https://arxiv.org/abs/2312.00220 · HTML: https://mlsanity.com/paper/2312.00220/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Contrastive Test-Time Adaptation](https://mlsanity.com/paper/2204.10377/) (2022)
- [TransNet V2: An effective deep network architecture for fast shot transition detection](https://mlsanity.com/paper/2008.04838/) (2020)
- [Boundary Proposal Network for Two-Stage Natural Language Video Localization](https://mlsanity.com/paper/2103.08109/) (2021)
- [Text Segmentation as a Supervised Learning Task](https://mlsanity.com/paper/1803.09337/) (2018)
- [A Local-to-Global Approach to Multi-modal Movie Scene Segmentation](https://mlsanity.com/paper/2004.02678/) (2020)
- [Text Segmentation by Cross Segment Attention](https://mlsanity.com/paper/2004.14535/) (2020)
- [Learning Cross-modal Contrastive Features for Video Domain Adaptation](https://mlsanity.com/paper/2108.11974/) (2021)
- [Two-Level Transformer and Auxiliary Coherence Modeling for Improved Text Segmentation](https://mlsanity.com/paper/2001.00891/) (2020)
- [Shot Contrastive Self-Supervised Learning for Scene Boundary Detection](https://mlsanity.com/paper/2104.13537/) (2021)
- [Improving Unsupervised Dialogue Topic Segmentation with Utterance-Pair Coherence Scoring](https://mlsanity.com/paper/2106.06719/) (2021)
- [Slow and steady feature analysis: higher order temporal coherence in video](https://mlsanity.com/paper/1506.04714/) (2015)
- [End-to-end Dense Video Captioning as Sequence Generation](https://mlsanity.com/paper/2204.08121/) (2022)
- [Transformer over Pre-trained Transformer for Neural Text Segmentation with Enhanced Topic Coherence](https://mlsanity.com/paper/2110.07160/) (2021)
- [A Deep Siamese Network for Scene Detection in Broadcast Videos](https://mlsanity.com/paper/1510.08893/) (2015)
- [Improving Context Modeling in Neural Topic Segmentation](https://mlsanity.com/paper/2010.03138/) (2020)
- [MCQA: Multimodal Co-attention Based Network for Question Answering](https://mlsanity.com/paper/2004.12238/) (2020)
- [Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment](https://mlsanity.com/paper/2210.04722/) (2022)
- [Contrastive Video-Language Learning with Fine-grained Frame Sampling](https://mlsanity.com/paper/2210.05039/) (2022)
- [Improving Topic Segmentation by Injecting Discourse Dependencies](https://mlsanity.com/paper/2209.08626/) (2022)
- [Multi-modal Video Chapter Generation](https://mlsanity.com/paper/2209.12694/) (2022)

## Abstract (arXiv)

Video topic segmentation unveils the coarse-grained semantic structure underlying videos and is essential for other video understanding tasks. Given the recent surge in multi-modal, relying solely on a single modality is arguably insufficient. On the other hand, prior solutions for similar tasks like video scene/shot segmentation cater to short videos with clear visual shifts but falter for long videos with subtle changes, such as livestreams. In this paper, we introduce a multi-modal video topic segmenter that utilizes both video transcripts and frames, bolstered by a cross-modal attention mechanism. Furthermore, we propose a dual-contrastive learning framework adhering to the unsupervised domain adaptation paradigm, enhancing our model's adaptability to longer, more semantically complex videos. Experiments on short and long video corpora demonstrate that our proposed solution, significantly surpasses baseline methods in terms of both accuracy and transferability, in both intra- and cross-domain settings.