# Ig3D: Integrating 3D Face Representations in Facial Expression Inference

arXiv: https://arxiv.org/abs/2408.16907 · HTML: https://mlsanity.com/paper/2408.16907/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 70 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 30 of 35)

- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [EMOCA: Emotion Driven Monocular Face Capture and Animation](https://mlsanity.com/paper/2204.11312/) (2022)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network](https://mlsanity.com/paper/1910.11111/) (2019)
- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis](https://mlsanity.com/paper/2404.04104/) (2024)
- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.00847/) (2022)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2312.05447/) (2023)
- [Towards Open Domain Text-Driven Synthesis of Multi-Person Motions](https://mlsanity.com/paper/2405.18483/) (2024)
- [Pre-training strategies and datasets for facial representation learning](https://mlsanity.com/paper/2103.16554/) (2021)
- [Robust Face Recognition via Multimodal Deep Face Representation](https://mlsanity.com/paper/1509.00244/) (2015)
- [Language-guided Human Motion Synthesis with Atomic Actions](https://mlsanity.com/paper/2308.09611/) (2023)
- [SignAvatar: Sign Language 3D Motion Reconstruction and Generation](https://mlsanity.com/paper/2405.07974/) (2024)
- [Representation Learning and Identity Adversarial Training for Facial Behavior Understanding](https://mlsanity.com/paper/2407.11243/) (2024)
- [An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild](https://mlsanity.com/paper/2107.03465/) (2021)
- [VEATIC: Video-based Emotion and Affect Tracking in Context Dataset](https://mlsanity.com/paper/2309.06745/) (2023)

## Built on by (most cited first, 6 of 6)

- [BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change](https://mlsanity.com/paper/2505.19328/) (2025)
- [Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions](https://mlsanity.com/paper/2604.11730/) (2026)
- [FIELDS: Face reconstruction with accurate Inference of Expression using Learning with Direct Supervision](https://mlsanity.com/paper/2511.21245/) (2025)
- [InterventionLens: A Multi-Agent Framework for Detecting ASD Intervention Strategies in Parent-Child Shared Reading](https://mlsanity.com/paper/2603.13710/) (2026)
- [ConfusionBench: An Expert-Validated Benchmark for Confusion Recognition and Localization in Educational Videos](https://mlsanity.com/paper/2603.17267/) (2026)
- [SuperFace: Preference-Aligned Facial Expression Estimation Beyond Pseudo Supervision](https://mlsanity.com/paper/2605.06179/) (2026)

## Abstract (arXiv)

Reconstructing 3D faces with facial geometry from single images has allowed for major advances in animation, generative models, and virtual reality. However, this ability to represent faces with their 3D features is not as fully explored by the facial expression inference (FEI) community. This study therefore aims to investigate the impacts of integrating such 3D representations into the FEI task, specifically for facial expression classification and face-based valence-arousal (VA) estimation. To accomplish this, we first assess the performance of two 3D face representations (both based on the 3D morphable model, FLAME) for the FEI tasks. We further explore two fusion architectures, intermediate fusion and late fusion, for integrating the 3D face representations with existing 2D inference frameworks. To evaluate our proposed architecture, we extract the corresponding 3D representations and perform extensive tests on the AffectNet and RAF-DB datasets. Our experimental results demonstrate that our proposed method outperforms the state-of-the-art AffectNet VA estimation and RAF-DB classification tasks. Moreover, our method can act as a complement to other existing methods to boost performance in many emotion inference tasks.