# FM-ViT: Flexible Modal Vision Transformers for Face Anti-Spoofing

arXiv: https://arxiv.org/abs/2305.03277 · HTML: https://mlsanity.com/paper/2305.03277/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 72 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [Searching Central Difference Convolutional Networks for Face Anti-Spoofing](https://mlsanity.com/paper/2003.04092/) (2020)
- [Learning Deep Models for Face Anti-Spoofing: Binary or Auxiliary Supervision](https://mlsanity.com/paper/1803.11097/) (2018)
- [CelebA-Spoof: Large-Scale Face Anti-Spoofing Dataset with Rich Annotations](https://mlsanity.com/paper/2007.12342/) (2020)
- [NAS-FAS: Static-Dynamic Central Difference Network Search for Face Anti-Spoofing](https://mlsanity.com/paper/2011.02062/) (2020)
- [Cross-domain Face Presentation Attack Detection via Multi-domain Disentangled Representation Learning](https://mlsanity.com/paper/2004.01959/) (2020)
- [face anti-spoofing based on color texture analysis](https://mlsanity.com/paper/1511.06316/) (2015)
- [Learn Convolutional Neural Network for Face Anti-Spoofing](https://mlsanity.com/paper/1408.5601/) (2014)
- [Biometric Face Presentation Attack Detection with Multi-Channel Convolutional Neural Network](https://mlsanity.com/paper/1909.08848/) (2019)
- [Deep Pixel-wise Binary Supervision for Face Presentation Attack Detection](https://mlsanity.com/paper/1907.04047/) (2019)
- [Deep Spatial Gradient and Temporal Depth Learning for Face Anti-spoofing](https://mlsanity.com/paper/2003.08061/) (2020)
- [PatchNet: A Simple Face Anti-Spoofing Framework via Fine-Grained Patch Recognition](https://mlsanity.com/paper/2203.14325/) (2022)
- [CASIA-SURF CeFA: A Benchmark for Multi-modal Cross-ethnicity Face Anti-spoofing](https://mlsanity.com/paper/2003.05136/) (2020)
- [Face Anti-Spoofing Via Disentangled Representation Learning](https://mlsanity.com/paper/2008.08250/) (2020)
- [KVT: k-NN Attention for Boosting Vision Transformers](https://mlsanity.com/paper/2106.00515/) (2021)
- [Face Anti-Spoofing with Human Material Perception](https://mlsanity.com/paper/2007.02157/) (2020)
- [On Disentangling Spoof Trace for Generic Face Anti-Spoofing](https://mlsanity.com/paper/2007.09273/) (2020)
- [CASIA-SURF: A Large-scale Multi-modal Benchmark for Face Anti-spoofing](https://mlsanity.com/paper/1908.10654/) (2019)
- [Face De-Spoofing: Anti-Spoofing via Noise Modeling](https://mlsanity.com/paper/1807.09968/) (2018)
- [A Dataset and Benchmark for Large-scale Multi-modal Face Anti-spoofing](https://mlsanity.com/paper/1812.00408/) (2018)

## Built on by (most cited first, 22 of 22)

- [Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection](https://mlsanity.com/paper/2312.16649/) (2023)
- [CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-spoofing](https://mlsanity.com/paper/2403.14333/) (2024)
- [Test-Time Domain Generalization for Face Anti-Spoofing](https://mlsanity.com/paper/2403.19334/) (2024)
- [Unified Physical-Digital Face Attack Detection](https://mlsanity.com/paper/2401.17699/) (2024)
- [Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation](https://mlsanity.com/paper/2309.09667/) (2023)
- [Visual Prompt Flexible-Modal Face Anti-Spoofing](https://mlsanity.com/paper/2307.13958/) (2023)
- [Joint Physical-Digital Facial Attack Detection Via Simulating Spoofing Clues](https://mlsanity.com/paper/2404.08450/) (2024)
- [La-SoftMoE CLIP for Unified Physical-Digital Face Attack Detection](https://mlsanity.com/paper/2408.12793/) (2024)
- [Domain Generalization for Face Anti-spoofing via Content-aware Composite Prompt Engineering](https://mlsanity.com/paper/2504.04470/) (2025)
- [Fine-Grained Annotation for Face Anti-Spoofing](https://mlsanity.com/paper/2310.08142/) (2023)
- [DADM: Dual Alignment of Domain and Modality for Face Anti-spoofing](https://mlsanity.com/paper/2503.00429/) (2025)
- [Unified Physical-Digital Attack Detection Challenge](https://mlsanity.com/paper/2404.06211/) (2024)
- [FA^{3}-CLIP: Frequency-Aware Cues Fusion and Attack-Agnostic Prompt Learning for Unified Face Attack Detection](https://mlsanity.com/paper/2504.00454/) (2025)
- [SUEDE:Shared Unified Experts for Physical-Digital Face Attack Detection Enhancement](https://mlsanity.com/paper/2504.04818/) (2025)
- [Liveness Detection in Computer Vision: Transformer-based Self-Supervised Learning for Face Anti-Spoofing](https://mlsanity.com/paper/2406.13860/) (2024)
- [PriorNet: Prior-Guided Engagement Estimation from Face Video](https://mlsanity.com/paper/2605.03615/) (2026)
- [CLIPC8: Face liveness detection algorithm based on image-text pairs and contrastive learning](https://mlsanity.com/paper/2311.17583/) (2023)
- [Spoof Trace Discovery for Deep Learning Based Explainable Face Anti-Spoofing](https://mlsanity.com/paper/2412.17541/) (2024)
- [Multi-Modal Face Anti-Spoofing via Cross-Modal Feature Transitions](https://mlsanity.com/paper/2507.05575/) (2025)
- [Robust Multi-Modal Face Anti-Spoofing with Domain Adaptation: Tackling Missing Modalities, Noisy Pseudo-Labels, and Model Degradation](https://mlsanity.com/paper/2509.23475/) (2025)
- [Spoofing-aware Prompt Learning for Unified Physical-Digital Facial Attack Detection](https://mlsanity.com/paper/2512.06363/) (2025)
- [Architectural Bias in Face Presentation Attack Detection: A Comparative Study of Vision Transformers and Convolutional Neural Networks](https://mlsanity.com/paper/2606.18510/) (2026)

## Abstract (arXiv)

The availability of handy multi-modal (i.e., RGB-D) sensors has brought about a surge of face anti-spoofing research. However, the current multi-modal face presentation attack detection (PAD) has two defects: (1) The framework based on multi-modal fusion requires providing modalities consistent with the training input, which seriously limits the deployment scenario. (2) The performance of ConvNet-based model on high fidelity datasets is increasingly limited. In this work, we present a pure transformer-based framework, dubbed the Flexible Modal Vision Transformer (FM-ViT), for face anti-spoofing to flexibly target any single-modal (i.e., RGB) attack scenarios with the help of available multi-modal data. Specifically, FM-ViT retains a specific branch for each modality to capture different modal information and introduces the Cross-Modal Transformer Block (CMTB), which consists of two cascaded attentions named Multi-headed Mutual-Attention (MMA) and Fusion-Attention (MFA) to guide each modal branch to mine potential features from informative patch tokens, and to learn modality-agnostic liveness features by enriching the modal information of own CLS token, respectively. Experiments demonstrate that the single model trained based on FM-ViT can not only flexibly evaluate different modal samples, but also outperforms existing single-modal frameworks by a large margin, and approaches the multi-modal frameworks introduced with smaller FLOPs and model parameters.