# Video Representation Learning Using Discriminative Pooling

arXiv: https://arxiv.org/abs/1803.10628 · HTML: https://mlsanity.com/paper/1803.10628/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Mar 2018.

## Built on by (most cited first, 7 of 7)

- [Max-Margin Contrastive Learning](https://mlsanity.com/paper/2112.11450/) (2021)
- [Condensing a Sequence to One Informative Frame for Video Recognition](https://mlsanity.com/paper/2201.04022/) (2022)
- [Busy-Quiet Video Disentangling for Video Classification](https://mlsanity.com/paper/2103.15584/) (2021)
- [Transformer-based end-to-end classification of variable-length volumetric data](https://mlsanity.com/paper/2307.06666/) (2023)
- [Generalized One-Class Learning Using Pairs of Complementary Classifiers](https://mlsanity.com/paper/2106.13272/) (2021)
- [Dynamic Appearance: A Video Representation for Action Recognition with Joint Training](https://mlsanity.com/paper/2211.12748/) (2022)
- [Factorized Latent Dynamics for Video JEPA: An Empirical Study of Auxiliary Objectives](https://mlsanity.com/paper/2605.17165/) (2026)

## Abstract (arXiv)

Popular deep models for action recognition in videos generate independent predictions for short clips, which are then pooled heuristically to assign an action label to the full video segment. As not all frames may characterize the underlying action---indeed, many are common across multiple actions---pooling schemes that impose equal importance on all frames might be unfavorable. In an attempt to tackle this problem, we propose discriminative pooling, based on the notion that among the deep features generated on all short clips, there is at least one that characterizes the action. To this end, we learn a (nonlinear) hyperplane that separates this unknown, yet discriminative, feature from the rest. Applying multiple instance learning in a large-margin setup, we use the parameters of this separating hyperplane as a descriptor for the full video segment. Since these parameters are directly related to the support vectors in a max-margin framework, they serve as robust representations for pooling of the features. We formulate a joint objective and an efficient solver that learns these hyperplanes per video and the corresponding action classifiers over the hyperplanes. Our pooling scheme is end-to-end trainable within a deep framework. We report results from experiments on three benchmark datasets spanning a variety of challenges and demonstrate state-of-the-art performance across these tasks.