# AI for human assessment: What do professional assessors need?

arXiv: https://arxiv.org/abs/2204.08471 · HTML: https://mlsanity.com/paper/2204.08471/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 24 references that are in the map.
- First posted on arXiv 18 Apr 2022.

## Builds on (most cited first, 5 of 5)

- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [REsCUE: A framework for REal-time feedback on behavioral CUEs using multimodal anomaly detection](https://mlsanity.com/paper/1903.11485/) (2019)
- [Mindless Attractor: A False-Positive Resistant Intervention for Drawing Attention Using Auditory Perturbation](https://mlsanity.com/paper/2101.08621/) (2021)
- [CalmResponses: Displaying Collective Audience Reactions in Remote Communication](https://mlsanity.com/paper/2204.02308/) (2022)

## Built on by (most cited first, 3 of 3)

- [CatAlyst: Domain-Extensible Intervention for Preventing Task Procrastination Using Large Generative Models](https://mlsanity.com/paper/2302.05678/) (2023)
- [Supporting Experts with a Multimodal Machine-Learning-Based Tool for Human Behavior Analysis of Conversational Videos](https://mlsanity.com/paper/2402.11145/) (2024)
- [PhoenixNest-Video: Evidence-Grounded Multimodal Agent Framework for Automated Video Interview Assessment](https://mlsanity.com/paper/2609.02231/) (2026)

## Abstract (arXiv)

Recent organizations have started to adopt AI-based decision support tools to optimize human resource development practices, while facing various challenges of using AIs in highly contextual and sensitive domains. We present our case study that aims to help professional assessors make decisions in human assessment, in which they conduct interviews with assessees and evaluate their suitability for certain job roles. Our workshop with two industrial assessors elucidated troubles they face (i.e., maintaining stable and non-subjective observation of assessees' behaviors) and derived requirements of AI systems (i.e., extracting their nonverbal cues from interview videos in an interpretable manner). In response, we employed an unsupervised anomaly detection algorithm using multimodal behavioral features such as facial keypoints, body and head pose, and gaze. The algorithm extracts outlier scenes from the video based on behavioral features as well as informing which feature contributes to the outlierness. We first evaluated how the assessors would perceive the extracted cues and discovered that the algorithm is useful in suggesting scenes to which assessors would pay attention, thanks to its interpretability. Then, we developed an interface prototype incorporating the algorithm and had six assessors use it for their actual assessment. Their comments revealed the effectiveness of introducing unsupervised anomaly detection to enhance their feeling of confidence and objectivity of the assessment along with potential use scenarios of such AI-based systems in human assessment. Our approach, which builds on top of the idea of separating observation and interpretation in human-AI collaboration, will facilitate human decision making in highly contextual domains, such as human assessment, while keeping their trust in the system.