mlsanityData to 7 Oct 2026

Collaboratively Self-supervised Video Representation Learning for Action Recognition

arXiv 2401.07584 ↗First posted 15 Jan 2024cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Considering the close connection between action recognition and human pose estimation, we design a Collaboratively Self-supervised Video Representation (CSVR) learning framework specific to action recognition by jointly factoring in generative pose prediction and discriminative context matching as pretext tasks. Specifically, our CSVR consists of three branches: a generative pose prediction branch, a discriminative context matching branch, and a video generating branch. Among them, the first one encodes dynamic motion feature by utilizing Conditional-GAN to predict the human poses of future frames, and the second branch extracts static context features by contrasting positive and negative video feature and I-frame feature pairs. The third branch is designed to generate both current and future video frames, for the purpose of collaboratively improving dynamic motion features and static context features. Extensive experiments demonstrate that our method achieves state-of-the-art performance on multiple popular video datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 137 of its 176 references that are in the map. First posted on arXiv 15 Jan 2024.

Builds on

137 of 176 references.

References · 137

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
38 more
  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  7. 2022Video Diffusion Modelscited by 2,280
  8. 2017Improved Training of Wasserstein GANscited by 2,223
  9. 2014Conditional Generative Adversarial Netscited by 2,152
  10. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  11. 2017The Kinetics Human Action Video Datasetcited by 1,894
  12. 2018SlowFast Networks for Video Recognitioncited by 1,862
  13. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  14. 2021ViViT: A Video Vision Transformercited by 1,610
  15. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  16. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  17. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  18. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  19. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  20. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  21. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  22. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  23. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  24. 2019Contrastive Multiview Codingcited by 1,297
  25. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  26. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  27. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  28. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  29. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  30. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  31. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  32. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  33. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  34. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  35. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  36. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  37. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  38. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521

50 most cited of 137 All 137 (JSON)

Built on by

Not cited yet.