# LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videos

arXiv: https://arxiv.org/abs/2210.05840 · HTML: https://mlsanity.com/paper/2210.05840/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 93 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [TransNet V2: An effective deep network architecture for fast shot transition detection](https://mlsanity.com/paper/2008.04838/) (2020)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Graph Optimal Transport for Cross-Domain Alignment](https://mlsanity.com/paper/2006.14744/) (2020)
- [A Local-to-Global Approach to Multi-modal Movie Scene Segmentation](https://mlsanity.com/paper/2004.02678/) (2020)
- [Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation](https://mlsanity.com/paper/2103.11264/) (2021)
- [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017)
- [UFO: A UniFied TransfOrmer for Vision-Language Representation Learning](https://mlsanity.com/paper/2111.10023/) (2021)
- [CO-Optimal Transport](https://mlsanity.com/paper/2002.03731/) (2020)
- [A Perceptual Prediction Framework for Self Supervised Event Segmentation](https://mlsanity.com/paper/1811.04869/) (2018)
- [Hierarchical Optimal Transport for Multimodal Distribution Alignment](https://mlsanity.com/paper/1906.11768/) (2019)
- [Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations](https://mlsanity.com/paper/1905.06139/) (2019)
- [A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation](https://mlsanity.com/paper/1906.01028/) (2019)
- [Shot Contrastive Self-Supervised Learning for Scene Boundary Detection](https://mlsanity.com/paper/2104.13537/) (2021)
- [To Click or Not To Click: Automatic Selection of Beautiful Thumbnails from Videos](https://mlsanity.com/paper/1609.01388/) (2016)
- [Improving Sequence-to-Sequence Learning via Optimal Transport](https://mlsanity.com/paper/1901.06283/) (2019)
- [Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)](https://mlsanity.com/paper/1906.10555/) (2019)
- [Learning to Segment Actions from Observation and Narration](https://mlsanity.com/paper/2005.03684/) (2020)
- [Making transport more robust and interpretable by moving data through a small number of anchor points](https://mlsanity.com/paper/2012.11589/) (2020)

## Built on by (most cited first, 6 of 6)

- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?](https://mlsanity.com/paper/2301.09017/) (2023)
- [MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos](https://mlsanity.com/paper/2306.04216/) (2023)
- [HierSum: A Global and Local Attention Mechanism for Video Summarization](https://mlsanity.com/paper/2504.18689/) (2025)
- [Multimodal Fusion and Coherence Modeling for Video Topic Segmentation](https://mlsanity.com/paper/2408.00365/) (2024)
- [VPD-100K: Towards Generalizable and Fine-grained Visual Privacy Protection](https://mlsanity.com/paper/2605.10229/) (2026)

## Abstract (arXiv)

Livestream videos have become a significant part of online learning, where design, digital marketing, creative painting, and other skills are taught by experienced experts in the sessions, making them valuable materials. However, Livestream tutorial videos are usually hours long, recorded, and uploaded to the Internet directly after the live sessions, making it hard for other people to catch up quickly. An outline will be a beneficial solution, which requires the video to be temporally segmented according to topics. In this work, we introduced a large Livestream video dataset named MultiLive, and formulated the temporal segmentation of the long Livestream videos (TSLLV) task. We propose LiveSeg, an unsupervised Livestream video temporal Segmentation solution, which takes advantage of multimodal features from different domains. Our method achieved a 16.8% F1-score performance improvement compared with the state-of-the-art method.