# End-to-End Compressed Video Representation Learning for Generic Event Boundary Detection

arXiv: https://arxiv.org/abs/2203.15336 · HTML: https://mlsanity.com/paper/2203.15336/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 50 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [BMN: Boundary-Matching Network for Temporal Action Proposal Generation](https://mlsanity.com/paper/1907.09702/) (2019)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Compressed Video Action Recognition](https://mlsanity.com/paper/1712.00636/) (2017)
- [Long-term Temporal Convolutions for Action Recognition](https://mlsanity.com/paper/1604.04494/) (2016)
- [Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation](https://mlsanity.com/paper/1602.02995/) (2016)
- [Generic Event Boundary Detection: A Benchmark for Event Segmentation](https://mlsanity.com/paper/2101.10511/) (2021)
- [Appearance-and-Relation Networks for Video Classification](https://mlsanity.com/paper/1711.09125/) (2017)
- [Token Shift Transformer for Video Classification](https://mlsanity.com/paper/2108.02432/) (2021)
- [Real-time Action Recognition with Enhanced Motion Vector CNNs](https://mlsanity.com/paper/1604.07669/) (2016)
- [End-to-End Learning of Motion Representation for Video Understanding](https://mlsanity.com/paper/1804.00413/) (2018)
- [ConvNet Architecture Search for Spatiotemporal Feature Learning](https://mlsanity.com/paper/1708.05038/) (2017)
- [DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition](https://mlsanity.com/paper/1901.03460/) (2019)
- [Multi-Fiber Networks for Video Recognition](https://mlsanity.com/paper/1807.11195/) (2018)
- [Self-supervised Video Representation Learning by Context and Motion Decoupling](https://mlsanity.com/paper/2104.00862/) (2021)

## Built on by (most cited first, 16 of 16)

- [You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos](https://mlsanity.com/paper/2303.07863/) (2023)
- [SMC++: Masked Learning of Unsupervised Video Semantic Compression](https://mlsanity.com/paper/2406.04765/) (2024)
- [Task-Aware Encoder Control for Deep Video Compression](https://mlsanity.com/paper/2404.04848/) (2024)
- [Online Generic Event Boundary Detection](https://mlsanity.com/paper/2510.06855/) (2025)
- [What's in the Flow? Exploiting Temporal Motion Cues for Unsupervised Generic Event Boundary Detection](https://mlsanity.com/paper/2404.18935/) (2024)
- [Fine-grained Dynamic Network for Generic Event Boundary Detection](https://mlsanity.com/paper/2407.04274/) (2024)
- [Motion Aware Self-Supervision for Generic Event Boundary Detection](https://mlsanity.com/paper/2210.05574/) (2022)
- [Local Compressed Video Stream Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2309.15431/) (2023)
- [EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation](https://mlsanity.com/paper/2607.10165/) (2026)
- [Generic Event Boundary Detection in Video with Pyramid Features](https://mlsanity.com/paper/2301.04288/) (2023)
- [Generic Event Boundary Detection via Denoising Diffusion](https://mlsanity.com/paper/2508.12084/) (2025)
- [Temporal2Seq: A Unified Framework for Temporal Video Understanding Tasks](https://mlsanity.com/paper/2409.18478/) (2024)
- [Structured Context Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2512.00475/) (2025)
- [IVAC-P2L: Leveraging Irregular Repetition Priors for Improving Video Action Counting](https://mlsanity.com/paper/2403.11959/) (2024)
- [Towards Bitstream-corrupted Harsh Visual Understanding: Through Bitstream Language Modeling as Robust Semantic Priors](https://mlsanity.com/paper/2608.21837/) (2026)
- [Video Codec-Guided Training-Free Token Compression for Video Large Language Models](https://mlsanity.com/paper/2610.12959/) (2026)

## Abstract (arXiv)

Generic event boundary detection aims to localize the generic, taxonomy-free event boundaries that segment videos into chunks. Existing methods typically require video frames to be decoded before feeding into the network, which demands considerable computational power and storage space. To that end, we propose a new end-to-end compressed video representation learning for event boundary detection that leverages the rich information in the compressed domain, i.e., RGB, motion vectors, residuals, and the internal group of pictures (GOP) structure, without fully decoding the video. Specifically, we first use the ConvNets to extract features of the I-frames in the GOPs. After that, a light-weight spatial-channel compressed encoder is designed to compute the feature representations of the P-frames based on the motion vectors, residuals and representations of their dependent I-frames. A temporal contrastive module is proposed to determine the event boundaries of video sequences. To remedy the ambiguities of annotations and speed up the training process, we use the Gaussian kernel to preprocess the ground-truth event boundaries. Extensive experiments conducted on the Kinetics-GEBD dataset demonstrate that the proposed method achieves comparable results to the state-of-the-art methods with 4.5× faster running speed.