# PainFormer: a Vision Foundation Model for Automatic Pain Assessment

arXiv: https://arxiv.org/abs/2505.01571 · HTML: https://mlsanity.com/paper/2505.01571/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 124 references that are in the map.
- First posted on arXiv 2 May 2025.

## Builds on (most cited first, 30 of 30)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [DigiFace-1M: 1 Million Digital Face Images for Face Recognition](https://mlsanity.com/paper/2210.02579/) (2022)
- [ECG Heartbeat Classification: A Deep Transferable Representation](https://mlsanity.com/paper/1805.00794/) (2018)
- [SFace: Privacy-friendly and Accurate Face Recognition using Synthetic Data](https://mlsanity.com/paper/2206.10520/) (2022)
- [An Image Patch is a Wave: Phase-Aware Vision MLP](https://mlsanity.com/paper/2111.12294/) (2021)
- [Digital Voicing of Silent Speech](https://mlsanity.com/paper/2010.02960/) (2020)
- [SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams](https://mlsanity.com/paper/2012.02961/) (2020)
- [Transformer Encoder with Multiscale Deep Learning for Pain Classification Using Physiological Signals](https://mlsanity.com/paper/2303.06845/) (2023)
- [Twins-PainViT: Towards a Modality-Agnostic Vision Transformer Framework for Multimodal Automatic Pain Assessment using Facial Videos and fNIRS](https://mlsanity.com/paper/2407.19809/) (2024)
- [Synthetic Thermal and RGB Videos for Automatic Pain Assessment utilizing a Vision-MLP Architecture](https://mlsanity.com/paper/2407.19811/) (2024)
- [A Full Transformer-based Framework for Automatic Pain Estimation using Videos](https://mlsanity.com/paper/2412.15095/) (2024)
- [Faces of Experimental Pain: Transferability of Deep Learned Heat Pain Features to Electrical Pain](https://mlsanity.com/paper/2406.11808/) (2024)
- [Multi-task Neural Networks for Pain Intensity Estimation using Electrocardiogram and Demographic Factors](https://mlsanity.com/paper/2407.19475/) (2024)
- [Transformer with Leveraged Masked Autoencoder for video-based Pain Assessment](https://mlsanity.com/paper/2409.05088/) (2024)
- [Non-contact Pain Recognition from Video Sequences with Remote Physiological Measurements Prediction](https://mlsanity.com/paper/2105.08822/) (2021)

## Built on by (most cited first, 10 of 10)

- [Efficient Pain Recognition via Respiration Signals: A Single Cross-Attention Transformer Multi-Window Fusion Pipeline](https://mlsanity.com/paper/2507.21886/) (2025)
- [Multi-Representation Diagrams for Pain Recognition: Integrating Various Electrodermal Activity Signals into a Single Image](https://mlsanity.com/paper/2507.21881/) (2025)
- [Tiny-BioMoE: a Lightweight Embedding Model for Biosignal Analysis](https://mlsanity.com/paper/2507.21875/) (2025)
- [A Lightweight Transformer for Pain Recognition from Brain Activity](https://mlsanity.com/paper/2604.16491/) (2026)
- [A Unified Tokenization Framework for Pain Recognition using Heterogeneous 3D Modalities](https://mlsanity.com/paper/2607.19716/) (2026)
- [ReFace: Reorganizing Facial Spatiotemporal Representations for Improved Pain Assessment](https://mlsanity.com/paper/2607.19722/) (2026)
- [Towards a Unified Modality-Agnostic Multimodal Framework for Cognitive Workload Assessment](https://mlsanity.com/paper/2609.20199/) (2026)
- [Beyond the Raw Waveform: Fusing Visual Representations of EDA for Stress Detection](https://mlsanity.com/paper/2609.22095/) (2026)
- [One-Block Transformer (1BT) for EEG-Based Cognitive Workload Assessment](https://mlsanity.com/paper/2605.00856/) (2026)
- [FUSE: Frame-Unified Stress Estimation from Facial Video](https://mlsanity.com/paper/2608.10442/) (2026)

## Abstract (arXiv)

Pain is a manifold condition that impacts a significant percentage of the population. Accurate and reliable pain evaluation for the people suffering is crucial to developing effective and advanced pain management protocols. Automatic pain assessment systems provide continuous monitoring and support decision-making processes, ultimately aiming to alleviate distress and prevent functionality decline. This study introduces PainFormer, a vision foundation model based on multi-task learning principles trained simultaneously on 14 tasks/datasets with a total of 10.9 million samples. Functioning as an embedding extractor for various input modalities, the foundation model provides feature representations to the Embedding-Mixer, a transformer-based module that performs the final pain assessment. Extensive experiments employing behavioral modalities - including RGB, synthetic thermal, and estimated depth videos - and physiological modalities such as ECG, EMG, GSR, and fNIRS revealed that PainFormer effectively extracts high-quality embeddings from diverse input modalities. The proposed framework is evaluated on two pain datasets, BioVid and AI4Pain, and directly compared to 75 different methodologies documented in the literature. Experiments conducted in unimodal and multimodal settings demonstrate state-of-the-art performances across modalities and pave the way toward general-purpose models for automatic pain assessment. The foundation model's architecture (code) and weights are available at: https://github.com/GkikasStefanos/PainFormer.