# Generic Event Boundary Detection: A Benchmark for Event Segmentation

arXiv: https://arxiv.org/abs/2101.10511 · HTML: https://mlsanity.com/paper/2101.10511/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 61 references that are in the map.
- First posted on arXiv 26 Jan 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [BSN: Boundary Sensitive Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1806.02964/) (2018)
- [Temporal Action Detection with Structured Segment Networks](https://mlsanity.com/paper/1704.06228/) (2017)
- [Rethinking the Faster R-CNN Architecture for Temporal Action Localization](https://mlsanity.com/paper/1804.07667/) (2018)
- [Video Representation Learning by Dense Predictive Coding](https://mlsanity.com/paper/1909.04656/) (2019)
- [Gaussian Temporal Awareness Networks for Action Localization](https://mlsanity.com/paper/1909.03877/) (2019)
- [Single Shot Temporal Action Detection](https://mlsanity.com/paper/1710.06236/) (2017)
- [CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos](https://mlsanity.com/paper/1703.01515/) (2017)
- [Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation](https://mlsanity.com/paper/1602.02995/) (2016)
- [Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment](https://mlsanity.com/paper/1803.10699/) (2018)
- [Connectionist Temporal Modeling for Weakly Supervised Action Labeling](https://mlsanity.com/paper/1607.08584/) (2016)
- [An end-to-end generative framework for video segmentation and recognition](https://mlsanity.com/paper/1509.01947/) (2015)
- [Weakly Supervised Energy-Based Learning for Action Segmentation](https://mlsanity.com/paper/1909.13155/) (2019)
- [Intra- and Inter-Action Understanding via Temporal Action Parsing](https://mlsanity.com/paper/2005.10229/) (2020)
- [A Perceptual Prediction Framework for Self Supervised Event Segmentation](https://mlsanity.com/paper/1811.04869/) (2018)
- [Temporal Context Network for Activity Localization in Videos](https://mlsanity.com/paper/1708.02349/) (2017)
- [Cascaded Boundary Regression for Temporal Action Detection](https://mlsanity.com/paper/1705.01180/) (2017)
- [Joint Discovery of Object States and Manipulation Actions](https://mlsanity.com/paper/1702.02738/) (2017)
- [Temporal Action Localization by Structured Maximal Sums](https://mlsanity.com/paper/1704.04671/) (2017)

## Built on by (most cited first, 30 of 65)

- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [MovieChat+: Question-aware Sparse Memory for Long Video Question Answering](https://mlsanity.com/paper/2404.17176/) (2024)
- [Equivariant Similarity for Vision-Language Foundation Models](https://mlsanity.com/paper/2303.14465/) (2023)
- [V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning](https://mlsanity.com/paper/2404.12353/) (2024)
- [Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval](https://mlsanity.com/paper/2407.15051/) (2024)
- [LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answering](https://mlsanity.com/paper/2312.04817/) (2023)
- [AutoShot: A Short Video Dataset and State-of-the-Art Shot Boundary Detection](https://mlsanity.com/paper/2304.06116/) (2023)
- [UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2111.14799/) (2021)
- [Progressive Attention on Multi-Level Dense Difference Maps for Generic Event Boundary Detection](https://mlsanity.com/paper/2112.04771/) (2021)
- [GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval](https://mlsanity.com/paper/2204.00486/) (2022)
- [Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding](https://mlsanity.com/paper/2403.16276/) (2024)
- [VADTree: Explainable Training-Free Video Anomaly Detection via Hierarchical Granularity-Aware Tree](https://mlsanity.com/paper/2510.22693/) (2025)
- [Boundary-aware Self-supervised Learning for Video Scene Segmentation](https://mlsanity.com/paper/2201.05277/) (2022)
- [End-to-End Compressed Video Representation Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2203.15336/) (2022)
- [Learning from Untrimmed Videos: Self-Supervised Video Representation Learning with Hierarchical Consistency](https://mlsanity.com/paper/2204.03017/) (2022)
- [UnweaveNet: Unweaving Activity Stories](https://mlsanity.com/paper/2112.10194/) (2021)
- [Winning the CVPR'2021 Kinetics-GEBD Challenge: Contrastive Learning Approach](https://mlsanity.com/paper/2106.11549/) (2021)
- [Generic Event Boundary Detection Challenge at CVPR 2021 Technical Report: Cascaded Temporal Attention Network (CASTANET)](https://mlsanity.com/paper/2107.00239/) (2021)
- [Temporal Perceiver: A General Architecture for Arbitrary Boundary Detection](https://mlsanity.com/paper/2203.00307/) (2022)
- [OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation](https://mlsanity.com/paper/2309.06276/) (2023)
- [Structured Context Transformer for Generic Event Boundary Detection](https://mlsanity.com/paper/2206.02985/) (2022)
- [CoSeg: Cognitively Inspired Unsupervised Generic Event Segmentation](https://mlsanity.com/paper/2109.15170/) (2021)
- [Rethinking the Architecture Design for Efficient Generic Event Boundary Detection](https://mlsanity.com/paper/2407.12622/) (2024)
- [Discerning Generic Event Boundaries in Long-Form Wild Videos](https://mlsanity.com/paper/2106.10090/) (2021)
- [FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understanding](https://mlsanity.com/paper/2511.00141/) (2025)

## Abstract (arXiv)

This paper presents a novel task together with a new benchmark for detecting generic, taxonomy-free event boundaries that segment a whole video into chunks. Conventional work in temporal video segmentation and action detection focuses on localizing pre-defined action categories and thus does not scale to generic videos. Cognitive Science has known since last century that humans consistently segment videos into meaningful temporal chunks. This segmentation happens naturally, without pre-defined event categories and without being explicitly asked to do so. Here, we repeat these cognitive experiments on mainstream CV datasets; with our novel annotation guideline which addresses the complexities of taxonomy-free event boundary annotation, we introduce the task of Generic Event Boundary Detection (GEBD) and the new benchmark Kinetics-GEBD. Our Kinetics-GEBD has the largest number of boundaries (e.g. 32 of ActivityNet, 8 of EPIC-Kitchens-100) which are in-the-wild, taxonomy-free, cover generic event change, and respect human perception diversity. We view GEBD as an important stepping stone towards understanding the video as a whole, and believe it has been previously neglected due to a lack of proper task definition and annotations. Through experiment and human study we demonstrate the value of the annotations. Further, we benchmark supervised and un-supervised GEBD approaches on the TAPOS dataset and our Kinetics-GEBD. We release our annotations and baseline codes at CVPR'21 LOVEU Challenge: https://sites.google.com/view/loveucvpr21.