# Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing

arXiv: https://arxiv.org/abs/2302.05744 · HTML: https://mlsanity.com/paper/2302.05744/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 67 references that are in the map.
- First posted on arXiv 11 Feb 2023.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Searching Central Difference Convolutional Networks for Face Anti-Spoofing](https://mlsanity.com/paper/2003.04092/) (2020)
- [Learning Deep Models for Face Anti-Spoofing: Binary or Auxiliary Supervision](https://mlsanity.com/paper/1803.11097/) (2018)
- [Convolutional Bypasses Are Better Vision Transformer Adapters](https://mlsanity.com/paper/2207.07039/) (2022)
- [PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer](https://mlsanity.com/paper/2111.12082/) (2021)
- [Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training](https://mlsanity.com/paper/2209.07098/) (2022)
- [face anti-spoofing based on color texture analysis](https://mlsanity.com/paper/1511.06316/) (2015)
- [Biometric Face Presentation Attack Detection with Multi-Channel Convolutional Neural Network](https://mlsanity.com/paper/1909.08848/) (2019)
- [Deep Pixel-wise Binary Supervision for Face Presentation Attack Detection](https://mlsanity.com/paper/1907.04047/) (2019)
- [CASIA-SURF CeFA: A Benchmark for Multi-modal Cross-ethnicity Face Anti-spoofing](https://mlsanity.com/paper/2003.05136/) (2020)
- [Face Anti-Spoofing with Human Material Perception](https://mlsanity.com/paper/2007.02157/) (2020)
- [Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets](https://mlsanity.com/paper/2208.07463/) (2022)
- [CASIA-SURF: A Large-scale Multi-modal Benchmark for Face Anti-spoofing](https://mlsanity.com/paper/1908.10654/) (2019)
- [A Dataset and Benchmark for Large-scale Multi-modal Face Anti-spoofing](https://mlsanity.com/paper/1812.00408/) (2018)
- [Dual-Cross Central Difference Network for Face Anti-Spoofing](https://mlsanity.com/paper/2105.01290/) (2021)
- [Learning Meta Model for Zero- and Few-shot Face Anti-spoofing](https://mlsanity.com/paper/1904.12490/) (2019)
- [On the Effectiveness of Vision Transformers for Zero-shot Face Anti-Spoofing](https://mlsanity.com/paper/2011.08019/) (2020)
- [Contrastive Context-Aware Learning for 3D High-Fidelity Mask Face Presentation Attack Detection](https://mlsanity.com/paper/2104.06148/) (2021)

## Built on by (most cited first, 19 of 19)

- [FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba](https://mlsanity.com/paper/2404.09498/) (2024)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning](https://mlsanity.com/paper/2412.12032/) (2024)
- [Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing](https://mlsanity.com/paper/2402.19298/) (2024)
- [Rehearsal-Free Domain Continual Face Anti-Spoofing: Generalize More and Forget Less](https://mlsanity.com/paper/2303.09914/) (2023)
- [Visual Prompt Flexible-Modal Face Anti-Spoofing](https://mlsanity.com/paper/2307.13958/) (2023)
- [FaceShield: Explainable Face Anti-Spoofing with Multimodal Large Language Models](https://mlsanity.com/paper/2505.09415/) (2025)
- [Spectraformer: A Unified Random Feature Framework for Transformer](https://mlsanity.com/paper/2405.15310/) (2024)
- [DADM: Dual Alignment of Domain and Modality for Face Anti-spoofing](https://mlsanity.com/paper/2503.00429/) (2025)
- [Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.13340/) (2026)
- [Multi-modal Document Presentation Attack Detection With Forensics Trace Disentanglement](https://mlsanity.com/paper/2404.06663/) (2024)
- [EnfoMax: Domain Entropy and Mutual Information Maximization for Domain Generalized Face Anti-spoofing](https://mlsanity.com/paper/2302.08674/) (2023)
- [Exploring the Adversarial Robustness of Face Forgery Detection with Decision-based Black-box Attacks](https://mlsanity.com/paper/2310.12017/) (2023)
- [Advancing Cross-Domain Generalizability in Face Anti-Spoofing: Insights, Design, and Metrics](https://mlsanity.com/paper/2406.12258/) (2024)
- [Multi-Modal Face Anti-Spoofing via Cross-Modal Feature Transitions](https://mlsanity.com/paper/2507.05575/) (2025)
- [Robust Multi-Modal Face Anti-Spoofing with Domain Adaptation: Tackling Missing Modalities, Noisy Pseudo-Labels, and Model Degradation](https://mlsanity.com/paper/2509.23475/) (2025)
- [Virtual camera detection: Catching video injection attacks in remote biometric systems](https://mlsanity.com/paper/2512.10653/) (2025)
- [Domain-generalizable Face Anti-Spoofing with Patch-based Multi-tasking and Artifact Pattern Conversion](https://mlsanity.com/paper/2604.09018/) (2026)
- [CardiacMamba: Fair and Robust RGB-RF Fusion for Remote Heart Rate Estimation via State Space Modeling](https://mlsanity.com/paper/2608.15831/) (2026)

## Abstract (arXiv)

Recently, vision transformer (ViT) based multimodal learning methods have been proposed to improve the robustness of face anti-spoofing (FAS) systems. However, there are still no works to explore the fundamental natures (textite.g., modality-aware inputs, suitable multimodal pre-training, and efficient finetuning) in vanilla ViT for multimodal FAS. In this paper, we investigate three key factors (i.e., inputs, pre-training, and finetuning) in ViT for multimodal FAS with RGB, Infrared (IR), and Depth. First, in terms of the ViT inputs, we find that leveraging local feature descriptors benefits the ViT on IR modality but not RGB or Depth modalities. Second, in observation of the inefficiency on direct finetuning the whole or partial ViT, we design an adaptive multimodal adapter (AMA), which can efficiently aggregate local multimodal features while freezing majority of ViT parameters. Finally, in consideration of the task (FAS vs. generic object classification) and modality (multimodal vs. unimodal) gaps, ImageNet pre-trained models might be sub-optimal for the multimodal FAS task. To bridge these gaps, we propose the modality-asymmetric masked autoencoder (M2A2E) for multimodal FAS self-supervised pre-training without costly annotated labels. Compared with the previous modality-symmetric autoencoder, the proposed M2A2E is able to learn more intrinsic task-aware representation and compatible with modality-agnostic (e.g., unimodal, bimodal, and trimodal) downstream settings. Extensive experiments with both unimodal (RGB, Depth, IR) and multimodal (RGB+Depth, RGB+IR, Depth+IR, RGB+Depth+IR) settings conducted on multimodal FAS benchmarks demonstrate the superior performance of the proposed methods. We hope these findings and solutions can facilitate the future research for ViT-based multimodal FAS.