# Twins-PainViT: Towards a Modality-Agnostic Vision Transformer Framework for Multimodal Automatic Pain Assessment using Facial Videos and fNIRS

arXiv: https://arxiv.org/abs/2407.19809 · HTML: https://mlsanity.com/paper/2407.19809/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 41 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Builds on (most cited first, 13 of 13)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Mobile-Former: Bridging MobileNet and Transformer](https://mlsanity.com/paper/2108.05895/) (2021)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention](https://mlsanity.com/paper/2305.07027/) (2023)
- [Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention](https://mlsanity.com/paper/2112.14000/) (2021)
- [Digital Voicing of Silent Speech](https://mlsanity.com/paper/2010.02960/) (2020)
- [Multi-task Neural Networks for Pain Intensity Estimation using Electrocardiogram and Demographic Factors](https://mlsanity.com/paper/2407.19475/) (2024)

## Built on by (most cited first, 14 of 14)

- [Efficient Pain Recognition via Respiration Signals: A Single Cross-Attention Transformer Multi-Window Fusion Pipeline](https://mlsanity.com/paper/2507.21886/) (2025)
- [PainFormer: a Vision Foundation Model for Automatic Pain Assessment](https://mlsanity.com/paper/2505.01571/) (2025)
- [Multi-Representation Diagrams for Pain Recognition: Integrating Various Electrodermal Activity Signals into a Single Image](https://mlsanity.com/paper/2507.21881/) (2025)
- [Tiny-BioMoE: a Lightweight Embedding Model for Biosignal Analysis](https://mlsanity.com/paper/2507.21875/) (2025)
- [Transformer with Leveraged Masked Autoencoder for video-based Pain Assessment](https://mlsanity.com/paper/2409.05088/) (2024)
- [A Lightweight Transformer for Pain Recognition from Brain Activity](https://mlsanity.com/paper/2604.16491/) (2026)
- [A Unified Tokenization Framework for Pain Recognition using Heterogeneous 3D Modalities](https://mlsanity.com/paper/2607.19716/) (2026)
- [ReFace: Reorganizing Facial Spatiotemporal Representations for Improved Pain Assessment](https://mlsanity.com/paper/2607.19722/) (2026)
- [Beyond the Raw Waveform: Fusing Visual Representations of EDA for Stress Detection](https://mlsanity.com/paper/2609.22095/) (2026)
- [A Multi-Scale Temporal Framework with Dynamic Fusion for EEG-Based Emotion Recognition](https://mlsanity.com/paper/2608.09088/) (2026)
- [One-Block Transformer (1BT) for EEG-Based Cognitive Workload Assessment](https://mlsanity.com/paper/2605.00856/) (2026)
- [FUSE: Frame-Unified Stress Estimation from Facial Video](https://mlsanity.com/paper/2608.10442/) (2026)
- [MUPA2E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment](https://mlsanity.com/paper/2608.15999/) (2026)
- [UNWIND: Any-Length Facial Video for Stress Detection without Temporal Windowing](https://mlsanity.com/paper/2609.29553/) (2026)

## Abstract (arXiv)

Automatic pain assessment plays a critical role for advancing healthcare and optimizing pain management strategies. This study has been submitted to the First Multimodal Sensing Grand Challenge for Next-Gen Pain Assessment (AI4PAIN). The proposed multimodal framework utilizes facial videos and fNIRS and presents a modality-agnostic approach, alleviating the need for domain-specific models. Employing a dual ViT configuration and adopting waveform representations for the fNIRS, as well as for the extracted embeddings from the two modalities, demonstrate the efficacy of the proposed method, achieving an accuracy of 46.76% in the multilevel pain assessment task.