# Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit

arXiv: https://arxiv.org/abs/2409.17717 · HTML: https://mlsanity.com/paper/2409.17717/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 63 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 30 of 36)

- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Characterizing and Avoiding Negative Transfer](https://mlsanity.com/paper/1811.09751/) (2018)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network](https://mlsanity.com/paper/1910.11111/) (2019)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [Deep Transparent Prediction through Latent Representation Analysis](https://mlsanity.com/paper/2009.07044/) (2020)
- [Learn From All: Erasing Attention Consistency for Noisy Label Facial Expression Recognition](https://mlsanity.com/paper/2207.10299/) (2022)
- [Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond](https://mlsanity.com/paper/2401.01219/) (2024)
- [Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment](https://mlsanity.com/paper/1803.05588/) (2018)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [A Deep Neural Architecture for Harmonizing 3-D Input Data Analysis and Decision Making in Medical Imaging](https://mlsanity.com/paper/2303.00175/) (2023)
- [Py-Feat: Python Facial Expression Analysis Toolbox](https://mlsanity.com/paper/2104.03509/) (2021)
- [MixAugment & Mixup: Augmentation Methods for Facial Expression Recognition](https://mlsanity.com/paper/2205.04442/) (2022)
- [Domain adaptation, Explainability & Fairness in AI for Medical Image Analysis: Diagnosis of COVID-19 based on 3-D Chest CT-scans](https://mlsanity.com/paper/2403.02192/) (2024)
- [Multi-modal Facial Affective Analysis based on Masked Autoencoder](https://mlsanity.com/paper/2303.10849/) (2023)
- [COVID-19 Computer-aided Diagnosis through AI-assisted CT Imaging Analysis: Deploying a Medical AI System](https://mlsanity.com/paper/2403.06242/) (2024)
- [Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition](https://mlsanity.com/paper/2303.08356/) (2023)
- [Identification of Negative Transfers in Multitask Learning Using Surrogate Models](https://mlsanity.com/paper/2303.14582/) (2023)
- [BTDNet: a Multi-Modal Approach for Brain Tumor Radiogenomic Classification](https://mlsanity.com/paper/2310.03485/) (2023)
- [Uncertainty-guided Contrastive Learning for Single Source Domain Generalisation](https://mlsanity.com/paper/2403.07514/) (2024)
- [SAM2CLIP2SAM: Vision Language Model for Segmentation of 3D CT Scans for Covid-19 Detection](https://mlsanity.com/paper/2407.15728/) (2024)
- [Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition](https://mlsanity.com/paper/2106.03487/) (2021)

## Built on by (most cited first, 16 of 16)

- [DVD: A Comprehensive Dataset for Advancing Violence Detection in Real-World Scenarios](https://mlsanity.com/paper/2506.05372/) (2025)
- [From Affect to Complex Behavior: Advancing Multimodal Human-Centered AI at the 10th ABAW Workshop & Competition](https://mlsanity.com/paper/2605.27451/) (2026)
- [Using Prompts to Guide Large Language Models in Imitating a Real Person's Language Style](https://mlsanity.com/paper/2410.03848/) (2024)
- [Multimodal Emotion Regression with Multi-Objective Optimization and VAD-Aware Audio Modeling for the 10th ABAW EMI Track](https://mlsanity.com/paper/2603.13760/) (2026)
- [Solution to the 10th ABAW Expression Recognition Challenge: A Robust Multimodal Framework with Safe Cross-Attention and Modality Dropout](https://mlsanity.com/paper/2603.08034/) (2026)
- [Stage-Adaptive Reliability Modeling for Continuous Valence-Arousal Estimation](https://mlsanity.com/paper/2603.11468/) (2026)
- [Multimodal Emotion Recognition via Bi-directional Cross-Attention and Temporal Modeling](https://mlsanity.com/paper/2603.11971/) (2026)
- [HSEmotion Team at ABAW-10 Competition: Facial Expression Recognition, Valence-Arousal Estimation, Action Unit Detection and Fine-Grained Violence Classification](https://mlsanity.com/paper/2603.12693/) (2026)
- [Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach](https://mlsanity.com/paper/2603.13056/) (2026)
- [Anchoring Emotions in Text: Robust Multimodal Fusion for Mimicry Intensity Estimation](https://mlsanity.com/paper/2603.14976/) (2026)
- [The Circumplex Degeneracy Behind the Rare-Class Limit in Affect Recognition](https://mlsanity.com/paper/2606.15763/) (2026)
- [A Shared Latent for Partially-Labeled Multi-Task Facial Affect Recognition](https://mlsanity.com/paper/2607.16285/) (2026)
- [Causal Supervision of Attention for Affective Behaviour Analysis](https://mlsanity.com/paper/2607.12091/) (2026)
- [AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow](https://mlsanity.com/paper/2607.13250/) (2026)
- [Strength-Parity Ensembling with Parameter-Isolated Experts for Multi-Task Affect Recognition](https://mlsanity.com/paper/2607.16290/) (2026)
- [AffectFuse: Cross-Task Feature Fusion with Temporal Modeling for Multi-Task Affective Behavior Analysis](https://mlsanity.com/paper/2607.16546/) (2026)

## Abstract (arXiv)

In this paper, we introduce Behavior4All, a comprehensive, open-source toolkit for in-the-wild facial behavior analysis, integrating Face Localization, Valence-Arousal Estimation, Basic Expression Recognition and Action Unit Detection, all within a single framework. Available in both CPU-only and GPU-accelerated versions, Behavior4All leverages 12 large-scale, in-the-wild datasets consisting of over 5 million images from diverse demographic groups. It introduces a novel framework that leverages distribution matching and label co-annotation to address tasks with non-overlapping annotations, encoding prior knowledge of their relatedness. In the largest study of its kind, Behavior4All outperforms both state-of-the-art and toolkits in overall performance as well as fairness across all databases and tasks. It also demonstrates superior generalizability on unseen databases and on compound expression recognition. Finally, Behavior4All is way times faster than other toolkits.