mlsanityData to 14 Oct 2026

MARLIN: Masked Autoencoder for facial video Representation LearnINg

arXiv 2211.06627 ↗First posted 12 Nov 2022cs.CVcited by 57

Not on the map. Why?

Abstract (arXiv)

This paper proposes a self-supervised approach to learn universal facial representations from videos, that can transfer across a variety of facial analysis tasks such as Facial Attribute Recognition (FAR), Facial Expression Recognition (FER), DeepFake Detection (DFD), and Lip Synchronization (LS). Our proposed framework, named MARLIN, is a facial video masked autoencoder, that learns highly robust and generic facial embeddings from abundantly available non-annotated web crawled facial videos. As a challenging auxiliary task, MARLIN reconstructs the spatio-temporal details of the face from the densely masked facial regions which mainly include eyes, nose, mouth, lips, and skin to capture local and global aspects that in turn help in encoding generic and transferable features. Through a variety of experiments on diverse downstream tasks, we demonstrate MARLIN to be an excellent facial video encoder as well as feature extractor, that performs consistently well across a variety of downstream tasks including FAR (1.13% gain over supervised benchmark), FER (2.64% gain over unsupervised benchmark), DFD (1.86% gain over unsupervised benchmark), LS (29.36% gain for Frechet Inception Distance), and even in low data regime. Our code and models are available at https://github.com/ControlNet/MARLIN .

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 61 of its 89 references that are in the map. First posted on arXiv 12 Nov 2022.

Builds on

61 of 89 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  11. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  12. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
38 more
  1. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  2. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  3. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  7. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  8. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,101
  9. 2019StarGAN v2: Diverse Image Synthesis for Multiple Domainscited by 1,098
  10. 2021Transformers in Vision: A Surveycited by 1,013
  11. 2017VGGFace2: A dataset for recognising faces across pose and agecited by 914
  12. 2021Multiscale Vision Transformerscited by 897
  13. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  14. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  15. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  16. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  17. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  18. 2016MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognitioncited by 624
  19. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  20. 2020Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Cluescited by 523
  21. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  22. 2019Face X-ray for More General Face Forgery Detectioncited by 470
  23. 2021Intriguing Properties of Vision Transformerscited by 446
  24. 2018MesoNet: a Compact Facial Video Forgery Detection Networkcited by 409
  25. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 387
  26. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  27. 2018Deep Facial Expression Recognition: A Surveycited by 270
  28. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 250
  29. 2019Large Scale Adversarial Representation Learningcited by 239
  30. 2020DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detectioncited by 233
  31. 2018DeepFakes: a New Threat to Face Recognition? Assessment and Detectioncited by 209
  32. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 195
  33. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  34. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  35. 2021M2TR: Multi-modal Multi-scale Transformers for Deepfake Detectioncited by 114
  36. 2021GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worldscited by 98
  37. 2020Towards Automatic Face-to-Face Translationcited by 96
  38. 2018Multi-attention Recurrent Network for Human Communication Comprehensioncited by 89

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 18; 2025: 23; 2026: 9.

57 papers.

Sort:

By area:

  1. 2024LAA-Net: Localized Artifact Attention Network for Quality-Agnostic and Generalizable Deepfake Detectioncited by 89
  2. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 80
  3. 2024AVFF: Audio-Visual Feature Fusion for Video Deepfake Detectioncited by 59
  4. 2023Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localizationcited by 43
  5. 2023From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videoscited by 39
  6. 2024Masked Image Modeling: A Surveycited by 29
  7. 2024FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learningcited by 23
  8. 2024SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scalecited by 19
  9. 2023LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysiscited by 18
  10. 2023Do I Have Your Attention: A Large Scale Engagement Prediction Dataset and Baselinescited by 17
  11. 2024Self-Supervised Facial Representation Learning with Facial Region Awarenesscited by 17
  12. 2024Representation Learning and Identity Adversarial Training for Facial Behavior Understandingcited by 14
38 more
  1. 2025Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuningcited by 11
  2. 2024A Timely Survey on Vision Transformer for Deepfake Detectioncited by 10
  3. 2024Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlookcited by 10
  4. 2025Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Reviewcited by 10
  5. 2024Progressive Representation Learning for Real-Time UAV Trackingcited by 9
  6. 2025UniGaze: Towards Universal Gaze Estimation via Large-scale Pre-Trainingcited by 8
  7. 2024Engagement Measurement Based on Facial Landmarks and Spatial-Temporal Graph Convolutional Networkscited by 7
  8. 2024Transformer with Leveraged Masked Autoencoder for video-based Pain Assessmentcited by 7
  9. 2025SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detectioncited by 7
  10. 2024Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challengescited by 6
  11. 2025SMARTe-VR: Student Monitoring and Adaptive Response Technology for e-Learning in Virtual Realitycited by 6
  12. 2024Task-adaptive Q-Facecited by 5
  13. 2023ProS: Facial Omni-Representation Learning via Prototype-based Self-Distillationcited by 3
  14. 2024JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generationcited by 3
  15. 2025PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognitioncited by 3
  16. 2024Hearing Loss Detection from Facial Expressions in One-on-one Conversationscited by 2
  17. 2025MVP: Multimodal Emotion Recognition based on Video and Physiological Signalscited by 2
  18. 2025V-NAW: Video-based Noise-aware Adaptive Weighting for Facial Expression Recognitioncited by 1
  19. 2025Trident: Detecting Face Forgeries with Adversarial Triplet Learningcited by 1
  20. 2025Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusioncited by 1
  21. 2025SyncLipMAE: Contrastive Masked Pretraining for Audio-Visual Talking-Face Representationcited by 1
  22. 2025Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservationcited by 1
  23. 2025Performance Decay in Deepfake Detection: The Limitations of Training on Outdated Datacited by 1
  24. 2026Leave No Stone Unturned: Uncovering Holistic Audio-Visual Intrinsic Coherence for Deepfake Detectioncited by 1
  25. 2023Private Gradient Estimation is Useful for Generative Modelingcited by 0
  26. 2024Real, fake and synthetic faces -- does the coin have three sides?cited by 0
  27. 2024Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognitioncited by 0
  28. 2025Experimenting with Affective Computing Models in Video Interviews with Spanish-speaking Older Adultscited by 0
  29. 2025PSG-MAE: Robust Multitask Sleep Event Monitoring using Multichannel PSG Reconstruction and Inter-channel Contrastive Learningcited by 0
  30. 2025Low-Rank Head Avatar Personalization with Registerscited by 0
  31. 2025EXPOTION: Facial Expression and Motion Control for Multimodal Music Generationcited by 0
  32. 2025OPEN: A Benchmark Dataset and Baseline for Older Adult Patient Engagement Recognition in Virtual Rehabilitation Learning Environmentscited by 0
  33. 2025Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systemscited by 0
  34. 2025Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voicecited by 0
  35. 2025Discrete Facial Encoding: : A Framework for Data-driven Facial Display Discoverycited by 0
  36. 2025Detecting AI-Generated Images via Contextual Anomaly Estimation in Masked AutoEncoderscited by 0
  37. 2025Beyond Flicker: Detecting Kinematic Inconsistencies for Generalizable Deepfake Video Detectioncited by 0
  38. 2026Fusion-SSAT: Unleashing the Potential of Self-supervised Auxiliary Task by Feature Fusion for Generalized Deepfake Detectioncited by 0

50 most cited of 57 All 57 (JSON)