# Don't Judge Me by My Face : An Indirect Adversarial Approach to Remove Sensitive Information From Multimodal Neural Representation in Asynchronous Job Video Interviews

arXiv: https://arxiv.org/abs/2110.09424 · HTML: https://mlsanity.com/paper/2110.09424/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 36 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices](https://mlsanity.com/paper/1906.09208/) (2019)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [What does it mean to solve the problem of discrimination in hiring? Social, technical and legal perspectives from the UK on automated hiring systems](https://mlsanity.com/paper/1910.06144/) (2019)
- [Privacy-preserving Voice Analysis via Disentangled Representations](https://mlsanity.com/paper/2007.15064/) (2020)
- [Privacy Enhanced Multimodal Neural Representations for Emotion Recognition](https://mlsanity.com/paper/1910.13212/) (2019)
- [Automated Analysis and Prediction of Job Interview Performance](https://mlsanity.com/paper/1504.03425/) (2015)
- [Explaining First Impressions: Modeling, Recognizing, and Explaining Apparent Personality from Videos](https://mlsanity.com/paper/1802.00745/) (2018)
- [HireNet: a Hierarchical Attention Model for the Automatic Analysis of Asynchronous Video Job Interviews](https://mlsanity.com/paper/1907.11062/) (2019)
- [Grading video interviews with fairness considerations](https://mlsanity.com/paper/2007.05461/) (2020)
- [To Trust, or Not to Trust? A Study of Human Bias in Automated Video Interview Assessments](https://mlsanity.com/paper/1911.13248/) (2019)

## Built on by (most cited first, 1 of 1)

- [Applications of Risk Science to AI Fairness Evaluation: Principles, Challenges, and Best Practices](https://mlsanity.com/paper/2608.29478/) (2026)

## Abstract (arXiv)

se of machine learning for automatic analysis of job interview videos has recently seen increased interest. Despite claims of fair output regarding sensitive information such as gender or ethnicity of the candidates, the current approaches rarely provide proof of unbiased decision-making, or that sensitive information is not used. Recently, adversarial methods have been proved to effectively remove sensitive information from the latent representation of neural networks. However, these methods rely on the use of explicitly labeled protected variables (e.g. gender), which cannot be collected in the context of recruiting in some countries (e.g. France). In this article, we propose a new adversarial approach to remove sensitive information from the latent representation of neural networks without the need to collect any sensitive variable. Using only a few frames of the interview, we train our model to not be able to find the face of the candidate related to the job interview in the inner layers of the model. This, in turn, allows us to remove relevant private information from these layers. Comparing our approach to a standard baseline on a public dataset with gender and ethnicity annotations, we show that it effectively removes sensitive information from the main network. Moreover, to the best of our knowledge, this is the first application of adversarial techniques for obtaining a multimodal fair representation in the context of video job interviews. In summary, our contributions aim at improving fairness of the upcoming automatic systems processing videos of job interviews for equality in job selection.