mlsanityData to 14 Oct 2026

Spatiotemporal Contrastive Learning of Facial Expressions in Videos

arXiv 2108.03064 ↗First posted 6 Aug 2021cs.CV cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

We propose a self-supervised contrastive learning approach for facial expression recognition (FER) in videos. We propose a novel temporal sampling-based augmentation scheme to be utilized in addition to standard spatial augmentations used for contrastive learning. Our proposed temporal augmentation scheme randomly picks from one of three temporal sampling techniques: (1) pure random sampling, (2) uniform sampling, and (3) sequential sampling. This is followed by a combination of up to three standard spatial augmentations. We then use a deep R(2+1)D network for FER, which we train in a self-supervised fashion based on the augmentations and subsequently fine-tune. Experiments are performed on the Oulu-CASIA dataset and the performance is compared to other works in FER. The results indicate that our method achieves an accuracy of 89.4%, setting a new state-of-the-art by outperforming other works. Additional experiments and analysis confirm the considerable contribution of the proposed temporal augmentation versus the existing spatial ones.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 29 of its 57 references that are in the map. First posted on arXiv 6 Aug 2021.

Builds on

29 of 57 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  7. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  8. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  9. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  10. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  11. 2016Colorful Image Colorizationcited by 1,114
  12. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
17 more
  1. 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 941
  2. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  3. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  4. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  5. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  6. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  7. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  8. 2018Tracking Emerges by Colorizing Videoscited by 166
  9. 2017Representation Learning by Learning to Countcited by 120
  10. 2019Multi-task Self-Supervised Learning for Human Activity Detectioncited by 102
  11. 2020Self-supervised ECG Representation Learning for Emotion Recognitioncited by 99
  12. 2019Self-supervised Learning for ECG-based Emotion Recognitioncited by 32
  13. 2016FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognitioncited by 25
  14. 2016Peak-Piloted Deep Network for Facial Expression Recognitioncited by 22
  15. 2018Instant Automated Inference of Perceived Mental Stress through Smartphone PPG and Thermal Imagingcited by 12
  16. 2019Classification of Hand Movements from EEG using a Deep Attention-based LSTM Networkcited by 12
  17. 2021CapsField: Light Field-based Face and Expression Recognition in the Wild using Capsule Routingcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3; 2023: 6.

11 papers.

  1. 2021Self-supervised Contrastive Learning of Multi-view Facial Expressionscited by 23
  2. 2023SVFAP: Self-supervised Video Facial Affect Perceivercited by 18
  3. 2021Face Trees for Expression Recognitioncited by 6
  4. 2022Analysis of Semi-Supervised Methods for Facial Expression Recognitioncited by 4
  5. 2022SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Videocited by 2
  6. 2023Efficient Labelling of Affective Video Datasets via Few-Shot & Multi-Task Contrastive Learningcited by 2
  7. 2023Contrastive Learning of View-Invariant Representations for Facial Expressions Recognitioncited by 2
  8. 2022Temporal Contrastive Learning with Curriculumcited by 1
  9. 2023Learning to Summarize Videos by Contrasting Clipscited by 1
  10. 2023Remote Heart Rate Monitoring in Smart Environments from Videos with Self-supervised Pre-trainingcited by 1
  11. 2023Active Learning with Contrastive Pre-training for Facial Expression Recognitioncited by 0