# Cross-Task Multi-Branch Vision Transformer for Facial Expression and Mask Wearing Classification

arXiv: https://arxiv.org/abs/2404.14606 · HTML: https://mlsanity.com/paper/2404.14606/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 22 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition](https://mlsanity.com/paper/1905.04075/) (2019)
- [NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space](https://mlsanity.com/paper/2309.14616/) (2023)
- [Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks](https://mlsanity.com/paper/2311.10944/) (2023)
- [Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review](https://mlsanity.com/paper/2402.10350/) (2024)
- [FaceX-Zoo: A PyTorch Toolbox for Face Recognition](https://mlsanity.com/paper/2101.04407/) (2021)
- [Exploring Diverse Methods in Visual Question Answering](https://mlsanity.com/paper/2404.13565/) (2024)
- [Contextual Hourglass Network for Semantic Segmentation of High Resolution Aerial Imagery](https://mlsanity.com/paper/1810.12813/) (2018)
- [MmAP : Multi-modal Alignment Prompt for Cross-domain Multi-task Learning](https://mlsanity.com/paper/2312.08636/) (2023)
- [Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Models](https://mlsanity.com/paper/2305.08776/) (2023)
- [Multimodal Semi-Supervised Learning for 3D Objects](https://mlsanity.com/paper/2110.11601/) (2021)
- [Particle Filter SLAM for Vehicle Localization](https://mlsanity.com/paper/2402.07429/) (2024)
- [News Recommendation with Attention Mechanism](https://mlsanity.com/paper/2402.07422/) (2024)
- [Rumor Detection with a novel graph neural network approach](https://mlsanity.com/paper/2403.16206/) (2024)
- [Class-Level Confidence Based 3D Semi-Supervised Learning](https://mlsanity.com/paper/2210.10138/) (2022)
- [Vehicle Pose and Shape Estimation through Multiple Monocular Vision](https://mlsanity.com/paper/1802.03515/) (2018)
- [Building Lane-Level Maps from Aerial Images](https://mlsanity.com/paper/2312.13449/) (2023)
- [Image Captioning in news report scenario](https://mlsanity.com/paper/2403.16209/) (2024)
- [The application of Augmented Reality (AR) in Remote Work and Education](https://mlsanity.com/paper/2404.10579/) (2024)
- [Deep Reinforcement Learning for Mobile Robot Path Planning](https://mlsanity.com/paper/2404.06974/) (2024)
- [Research on emotionally intelligent dialogue generation based on automatic dialogue system](https://mlsanity.com/paper/2404.11447/) (2024)

## Built on by (most cited first, 16 of 16)

- [Harnessing XGBoost for Robust Biomarker Selection of Obsessive-Compulsive Disorder (OCD) from Adolescent Brain Cognitive Development (ABCD) data](https://mlsanity.com/paper/2407.00028/) (2024)
- [RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Texts](https://mlsanity.com/paper/2405.13179/) (2024)
- [Enhance Image-to-Image Generation with LLaVA-generated Prompts](https://mlsanity.com/paper/2406.01956/) (2024)
- [Image anomaly detection and prediction scheme based on SSA optimized ResNet50-BiGRU model](https://mlsanity.com/paper/2406.13987/) (2024)
- [Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks](https://mlsanity.com/paper/2405.11704/) (2024)
- [Exploration of Attention Mechanism-Enhanced Deep Learning Models in the Mining of Medical Textual Data](https://mlsanity.com/paper/2406.00016/) (2024)
- [Exploration of Multi-Scale Image Fusion Systems in Intelligent Medical Image Analysis](https://mlsanity.com/paper/2406.18548/) (2024)
- [Multi-Modal UAV Detection, Classification and Tracking Algorithm -- Technical Report for CVPR 2024 UG2 Challenge](https://mlsanity.com/paper/2405.16464/) (2024)
- [Advanced AI Framework for Enhanced Detection and Assessment of Abdominal Trauma: Integrating 3D Segmentation with 2D CNN and RNN Models](https://mlsanity.com/paper/2407.16165/) (2024)
- [Applying Conditional Generative Adversarial Networks for Imaging Diagnosis](https://mlsanity.com/paper/2408.02074/) (2024)
- [Recording Brain Activity While Listening to Music Using Wearable EEG Devices Combined with Bidirectional Long Short-Term Memory Networks](https://mlsanity.com/paper/2408.12124/) (2024)
- [Advancements in Feature Extraction Recognition of Medical Imaging Systems Through Deep Learning Technique](https://mlsanity.com/paper/2406.18549/) (2024)
- [Affective Behaviour Analysis via Progressive Learning](https://mlsanity.com/paper/2407.16945/) (2024)
- [OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis](https://mlsanity.com/paper/2506.02891/) (2025)
- [Advanced Multimodal Deep Learning Architecture for Image-Text Matching](https://mlsanity.com/paper/2406.15306/) (2024)
- [Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Model](https://mlsanity.com/paper/2411.10863/) (2024)

## Abstract (arXiv)

With wearing masks becoming a new cultural norm, facial expression recognition (FER) while taking masks into account has become a significant challenge. In this paper, we propose a unified multi-branch vision transformer for facial expression recognition and mask wearing classification tasks. Our approach extracts shared features for both tasks using a dual-branch architecture that obtains multi-scale feature representations. Furthermore, we propose a cross-task fusion phase that processes tokens for each task with separate branches, while exchanging information using a cross attention module. Our proposed framework reduces the overall complexity compared with using separate networks for both tasks by the simple yet effective cross-task fusion phase. Extensive experiments demonstrate that our proposed model performs better than or on par with different state-of-the-art methods on both facial expression recognition and facial mask wearing classification task.