# Learning Video Representations using Contrastive Bidirectional Transformer

arXiv: https://arxiv.org/abs/1906.05743 · HTML: https://mlsanity.com/paper/1906.05743/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2019.

## Built on by (most cited first, 30 of 39)

- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers](https://mlsanity.com/paper/2106.05392/) (2021)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [Scaling Up Vision-Language Pre-training for Image Captioning](https://mlsanity.com/paper/2111.12233/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding](https://mlsanity.com/paper/2105.09996/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2105.10311/) (2021)
- [Divide and Contrast: Self-supervised Learning from Uncurated Data](https://mlsanity.com/paper/2105.08054/) (2021)
- [Robust Audio-Visual Instance Discrimination](https://mlsanity.com/paper/2103.15916/) (2021)
- [Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training](https://mlsanity.com/paper/2210.07688/) (2022)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [Learning the Predictability of the Future](https://mlsanity.com/paper/2101.01600/) (2021)
- [Playing Lottery Tickets with Vision and Language](https://mlsanity.com/paper/2104.11832/) (2021)
- [Representation Learning via Global Temporal Alignment and Cycle-Consistency](https://mlsanity.com/paper/2105.05217/) (2021)
- [Time-Equivariant Contrastive Video Representation Learning](https://mlsanity.com/paper/2112.03624/) (2021)
- [Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning](https://mlsanity.com/paper/2103.10211/) (2021)
- [Learning to Answer Visual Questions from Web Videos](https://mlsanity.com/paper/2205.05019/) (2022)
- [Self-supervised object detection from audio-visual correspondence](https://mlsanity.com/paper/2104.06401/) (2021)
- [Which Mutual-Information Representation Learning Objectives are Sufficient for Control?](https://mlsanity.com/paper/2106.07278/) (2021)
- [Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation](https://mlsanity.com/paper/2301.09209/) (2023)
- [Unsupervised Visual Representation Learning by Tracking Patches in Video](https://mlsanity.com/paper/2105.02545/) (2021)
- [Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos](https://mlsanity.com/paper/2110.10596/) (2021)
- [GEM: A General Evaluation Benchmark for Multimodal Tasks](https://mlsanity.com/paper/2106.09889/) (2021)
- [RepsNet: Combining Vision with Language for Automated Medical Reports](https://mlsanity.com/paper/2209.13171/) (2022)
- [Composable Augmentation Encoding for Video Representation Learning](https://mlsanity.com/paper/2104.00616/) (2021)
- [Unlocking Exocentric Video-Language Data for Egocentric Video Representation Learning](https://mlsanity.com/paper/2408.03567/) (2024)

## Abstract (arXiv)

This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.