mlsanityData to 14 Oct 2026

MV2MAE: Multi-View Video Masked Autoencoders

arXiv 2401.15900 ↗First posted 29 Jan 2024cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Videos captured from multiple viewpoints can help in perceiving the 3D structure of the world and benefit computer vision tasks such as action recognition, tracking, etc. In this paper, we present a method for self-supervised learning from synchronized multi-view videos. We use a cross-view reconstruction task to inject geometry information in the model. Our approach is based on the masked autoencoder (MAE) framework. In addition to the same-view decoder, we introduce a separate cross-view decoder which leverages cross-attention mechanism to reconstruct a target viewpoint video using a video from source viewpoint, to help representations robust to viewpoint changes. For videos, static regions can be reconstructed trivially which hinders learning meaningful representations. To tackle this, we introduce a motion-weighted reconstruction loss which improves temporal modeling. We report state-of-the-art results on the NTU-60, NTU-120 and ETRI datasets, as well as in the transfer learning setting on NUCLA, PKU-MMD-II and ROCOG-v2 datasets, demonstrating the robustness of our approach. Code will be made available.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 51 of its 71 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

51 of 71 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  6. 2017The Kinetics Human Action Video Datasetcited by 1,909
  7. 2021ViViT: A Video Vision Transformercited by 1,620
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  10. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  11. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 686
  12. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
38 more
  1. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
  2. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  3. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  4. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  5. 2020Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognitioncited by 343
  6. 2021Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognitioncited by 322
  7. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  8. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  9. 2019WoodScape: A multi-task, multi-camera fisheye dataset for autonomous drivingcited by 144
  10. 2020Self-supervised Video Representation Learning by Pace Predictioncited by 139
  11. 2014Cross-view Action Modeling, Learning and Recognitioncited by 134
  12. 2017PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understandingcited by 101
  13. 2019Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statisticscited by 92
  14. 2020MS2L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognitioncited by 92
  15. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  16. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  17. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 80
  18. 2021CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representationscited by 72
  19. 2018Self-Supervised Spatiotemporal Feature Learning via Video Rotation Predictioncited by 71
  20. 2021Skeleton Cloud Colorization for Unsupervised 3D Action Representation Learningcited by 58
  21. 2020VPN: Learning Video-Pose Embedding for Activities of Daily Livingcited by 57
  22. 2020MEVA: A Large-Scale Multiview, Multimodal Video Dataset for Activity Detectioncited by 44
  23. 2023MGMAE: Motion Guided Masking for Video Masked Autoencodingcited by 41
  24. 2018Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognitioncited by 40
  25. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  26. 2022CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillationcited by 38
  27. 2018Glimpse Clouds: Human Activity Recognition from Unstructured Feature Pointscited by 33
  28. 2018Unsupervised Learning of View-invariant Action Representationscited by 32
  29. 2019DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognitioncited by 30
  30. 2020ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applicationscited by 29
  31. 2020ETRI-Activity3D: A Large-Scale RGB-D Dataset for Robots to Recognize Daily Activities of the Elderlycited by 26
  32. 2023HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actionscited by 25
  33. 2021Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognitioncited by 23
  34. 2017Human Action Recognition: Pose-based Attention draws focus to Handscited by 21
  35. 2021ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpointscited by 20
  36. 2021Recognizing Actions in Videos from Unseen Viewpointscited by 18
  37. 2023Synthetic-to-Real Domain Adaptation for Action Recognition: A Dataset and Baseline Performancescited by 16
  38. 2018Adding Attentiveness to the Neurons in Recurrent Neural Networkscited by 11

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025MuM: Multi-View Masked Image Modeling for 3D Visioncited by 5
  2. 2025Content-Distortion High-Order Interaction for Blind Image Quality Assessmentcited by 2
  3. 2025VET-DINO: Learning Anatomical Understanding Through Multi-View Distillation in Veterinary Imagingcited by 1
  4. 2025Multi Camera Connected Vision System with Multi View Analytics: A Comprehensive Surveycited by 1
  5. 2025Towards Generalizing Temporal Action Segmentation to Unseen Viewscited by 0
  6. 2026SHANDS: A Multi-View Dataset and Benchmark for Surgical Hand-Gesture and Error Recognition Toward Medical Trainingcited by 0