# BEE-NET: A deep neural network to identify in-the-wild Bodily Expression of Emotions

arXiv: https://arxiv.org/abs/2402.13955 · HTML: https://mlsanity.com/paper/2402.13955/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 25 of 25)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning](https://mlsanity.com/paper/1802.09232/) (2018)
- [Timeception for Complex Action Recognition](https://mlsanity.com/paper/1812.01289/) (2018)
- [Context Based Emotion Recognition using EMOTIC Dataset](https://mlsanity.com/paper/2003.13401/) (2020)
- [Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach](https://mlsanity.com/paper/1810.11547/) (2018)
- [Unsupervised Person Re-identification by Soft Multilabel Learning](https://mlsanity.com/paper/1903.06325/) (2019)
- [Survey on Emotional Body Gesture Recognition](https://mlsanity.com/paper/1801.07481/) (2018)
- [Going Deeper in Facial Expression Recognition using Deep Neural Networks](https://mlsanity.com/paper/1511.04110/) (2015)
- [ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wild](https://mlsanity.com/paper/1808.09568/) (2018)
- [Actor Conditioned Attention Maps for Video Action Detection](https://mlsanity.com/paper/1812.11631/) (2018)
- [Noisy Student Training using Body Language Dataset Improves Facial Expression Recognition](https://mlsanity.com/paper/2008.02655/) (2020)
- [A multi-stream convolutional neural network for classification of progressive MCI in Alzheimer's disease using structural MRI images](https://mlsanity.com/paper/2203.01944/) (2022)
- [Pain level and pain-related behaviour classification using GRU-based sparsely-connected RNNs](https://mlsanity.com/paper/2212.14806/) (2022)

## Built on by (most cited first, 1 of 1)

- [L-SFAN: Lightweight Spatially-focused Attention Network for Pain Behavior Detection](https://mlsanity.com/paper/2406.16913/) (2024)

## Abstract (arXiv)

In this study, we investigate how environmental factors, specifically the scenes and objects involved, can affect the expression of emotions through body language. To this end, we introduce a novel multi-stream deep convolutional neural network named BEE-NET. We also propose a new late fusion strategy that incorporates meta-information on places and objects as prior knowledge in the learning process. Our proposed probabilistic pooling model leverages this information to generate a joint probability distribution of both available and anticipated non-available contextual information in latent space. Importantly, our fusion strategy is differentiable, allowing for end-to-end training and capturing of hidden associations among data points without requiring further post-processing or regularisation. To evaluate our deep model, we use the Body Language Database (BoLD), which is currently the largest available database for the Automatic Identification of the in-the-wild Bodily Expression of Emotions (AIBEE). Our experimental results demonstrate that our proposed approach surpasses the current state-of-the-art in AIBEE by a margin of 2.07%, achieving an Emotional Recognition Score of 66.33%.