# Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment

arXiv: https://arxiv.org/abs/2302.10908 · HTML: https://mlsanity.com/paper/2302.10908/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 115 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [One pixel attack for fooling deep neural networks](https://mlsanity.com/paper/1710.08864/) (2017)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [Synthesizing the preferred inputs for neurons in neural networks via deep generator networks](https://mlsanity.com/paper/1605.09304/) (2016)
- [Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices](https://mlsanity.com/paper/1906.09208/) (2019)

## Built on by (most cited first, 30 of 33)

- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape](https://mlsanity.com/paper/2312.10868/) (2023)
- [M2LADS: A System for Generating MultiModal Learning Analytics Dashboards in Open Education](https://mlsanity.com/paper/2305.12561/) (2023)
- [Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs](https://mlsanity.com/paper/2306.02864/) (2023)
- [Is My Data in Your AI? Membership Inference Test (MINT) applied to Face Biometrics](https://mlsanity.com/paper/2402.09225/) (2024)
- [Addressing Bias in LLMs: Strategies and Application to Fair AI-based Recruitment](https://mlsanity.com/paper/2506.11880/) (2025)
- [PBa-LLM: Privacy- and Bias-aware NLP using Named-Entity Recognition (NER)](https://mlsanity.com/paper/2507.02966/) (2025)
- [Is My Text in Your AI Model? Gradient-based Membership Inference Test applied to LLMs](https://mlsanity.com/paper/2503.07384/) (2025)
- [mEBAL2 Database and Benchmark: Image-based Multispectral Eyeblink Detection](https://mlsanity.com/paper/2309.07880/) (2023)
- [Benchmarking Graph Neural Networks for Document Layout Analysis in Public Affairs](https://mlsanity.com/paper/2505.14699/) (2025)
- [Is My Vision-Language Data in Your AI? Membership Inference Test (MINT) Demo 2](https://mlsanity.com/paper/2606.14748/) (2026)
- [SMARTe-VR: Student Monitoring and Adaptive Response Technology for e-Learning in Virtual Reality](https://mlsanity.com/paper/2501.10977/) (2025)
- [AI-based Multimodal Biometrics for Detecting Smartphone Distractions: Application to Online Learning](https://mlsanity.com/paper/2506.17364/) (2025)
- [Auditing Training Data in Domain-adapted LLMs: LoRA-MINT](https://mlsanity.com/paper/2606.06946/) (2026)
- [Overview of Risk Assessment and Management for Intelligent Systems under the AI Act and Beyond](https://mlsanity.com/paper/2607.02197/) (2026)
- [Is It Really You? Exploring Biometric Verification Scenarios in Photorealistic Talking-Head Avatar Videos](https://mlsanity.com/paper/2508.00748/) (2025)
- [DSO: Direct Steering Optimization for Bias Mitigation](https://mlsanity.com/paper/2512.15926/) (2025)
- [Membership Inference Test: Auditing Training Data in Object Classification Models](https://mlsanity.com/paper/2601.12929/) (2026)
- [Document Layout Annotation: Database and Benchmark in the Domain of Public Affairs](https://mlsanity.com/paper/2306.10046/) (2023)
- [SaFL: Sybil-aware Federated Learning with Application to Face Recognition](https://mlsanity.com/paper/2311.04346/) (2023)
- [Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdb](https://mlsanity.com/paper/2407.16892/) (2024)
- [Comprehensive Equity Index (CEI): Definition and Application to Bias Evaluation in Biometrics](https://mlsanity.com/paper/2409.01928/) (2024)
- [VideoRun2D: Cost-Effective Markerless Motion Capture for Sprint Biomechanics](https://mlsanity.com/paper/2409.10175/) (2024)
- [Balancing Tails when Comparing Distributions: Comprehensive Equity Index (CEI) with Application to Bias Evaluation in Operational Face Biometrics](https://mlsanity.com/paper/2506.10564/) (2025)
- [Synthetic CVs To Build and Test Fairness-Aware Hiring Tools](https://mlsanity.com/paper/2508.21179/) (2025)
- [More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Models](https://mlsanity.com/paper/2412.17505/) (2024)
- [FairSSL: Fair Multimodal Self-Supervised Learning](https://mlsanity.com/paper/2508.16748/) (2025)
- [A Data-to-Product Multimodal Conceptual Framework to Achieve Automated Software Evolution for Context-rich Intelligent Applications](https://mlsanity.com/paper/2404.04821/) (2024)
- [Using AI Alignment Theory to understand the potential pitfalls of regulatory frameworks](https://mlsanity.com/paper/2410.19749/) (2024)
- [Exploring Deep Learning and Ultra-Widefield Imaging for Diabetic Retinopathy and Macular Edema](https://mlsanity.com/paper/2603.08235/) (2026)

## Abstract (arXiv)

The presence of decision-making algorithms in society is rapidly increasing nowadays, while concerns about their transparency and the possibility of these algorithms becoming new sources of discrimination are arising. There is a certain consensus about the need to develop AI applications with a Human-Centric approach. Human-Centric Machine Learning needs to be developed based on four main requirements: (i) utility and social good; (ii) privacy and data ownership; (iii) transparency and accountability; and (iv) fairness in AI-driven decision-making processes. All these four Human-Centric requirements are closely related to each other. With the aim of studying how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data, we propose a fictitious case study focused on automated recruitment: FairCVtest. We train automatic recruitment algorithms using a set of multimodal synthetic profiles including image, text, and structured data, which are consciously scored with gender and racial biases. FairCVtest shows the capacity of the Artificial Intelligence (AI) behind automatic recruitment tools built this way (a common practice in many other application scenarios beyond recruitment) to extract sensitive information from unstructured data and exploit it in combination to data biases in undesirable (unfair) ways. We present an overview of recent works developing techniques capable of removing sensitive information and biases from the decision-making process of deep learning architectures, as well as commonly used databases for fairness research in AI. We demonstrate how learning approaches developed to guarantee privacy in latent spaces can lead to unbiased and fair automatic decision-making process.