# An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring

arXiv: https://arxiv.org/abs/2201.09151 · HTML: https://mlsanity.com/paper/2201.09151/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 98 references that are in the map.
- First posted on arXiv 23 Jan 2022.

## Builds on (most cited first, 12 of 12)

- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices](https://mlsanity.com/paper/1906.09208/) (2019)
- [The Ethics of AI Ethics -- An Evaluation of Guidelines](https://mlsanity.com/paper/1903.03425/) (2019)
- [Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination](https://mlsanity.com/paper/1408.6491/) (2014)
- [Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies](https://mlsanity.com/paper/2109.06974/) (2021)
- [Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits](https://mlsanity.com/paper/2102.04256/) (2021)
- [Auditing ML Models for Individual Bias and Unfairness](https://mlsanity.com/paper/2003.05048/) (2020)
- [A Silicon Valley Love Triangle: Hiring Algorithms, Pseudo-Science, and the Quest for Auditability](https://mlsanity.com/paper/2106.12403/) (2021)
- [Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations](https://mlsanity.com/paper/2110.10980/) (2021)
- [Does Fair Ranking Improve Minority Outcomes? Understanding the Interplay of Human and Algorithmic Biases in Online Hiring](https://mlsanity.com/paper/2012.00423/) (2020)
- [Hiring Fairly in the Age of Algorithms](https://mlsanity.com/paper/2004.07132/) (2020)

## Built on by (most cited first, 10 of 10)

- [Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey](https://mlsanity.com/paper/2309.13933/) (2023)
- [Auditing of AI: Legal, Ethical and Technical Approaches](https://mlsanity.com/paper/2407.06235/) (2024)
- [A Brief Tutorial on Sample Size Calculations for Fairness Audits](https://mlsanity.com/paper/2312.04745/) (2023)
- [Algorithmic Monocultures in Hiring](https://mlsanity.com/paper/2605.27371/) (2026)
- [The Cadaver in the Machine: The Social Practices of Measurement and Validation in Motion Capture Technology](https://mlsanity.com/paper/2401.10877/) (2024)
- [Still More Shades of Null: An Evaluation Suite for Responsible Missing Value Imputation](https://mlsanity.com/paper/2409.07510/) (2024)
- [Greater accessibility can amplify discrimination in generative AI](https://mlsanity.com/paper/2603.22260/) (2026)
- [The Initial Screening Order Problem](https://mlsanity.com/paper/2307.15398/) (2023)
- [White-Box Sensitivity Auditing with Steering Vectors](https://mlsanity.com/paper/2601.16398/) (2026)
- [Taking the Whys Seriously: Limitations of Counterfactual Explanations in Justification and Recourse](https://mlsanity.com/paper/2608.30956/) (2026)

## Abstract (arXiv)

Automated hiring systems are among the fastest-developing of all high-stakes AI systems. Among these are algorithmic personality tests that use insights from psychometric testing, and promise to surface personality traits indicative of future success based on job seekers' resumes or social media profiles. We interrogate the validity of such systems using stability of the outputs they produce, noting that reliability is a necessary, but not a sufficient, condition for validity. Our approach is to (a) develop a methodology for an external audit of stability of predictions made by algorithmic personality tests, and (b) instantiate this methodology in an audit of two systems, Humantic AI and Crystal. Crucially, rather than challenging or affirming the assumptions made in psychometric testing -- that personality is a meaningful and measurable construct, and that personality traits are indicative of future success on the job -- we frame our methodology around testing the underlying assumptions made by the vendors of the algorithmic personality tests themselves. Our main contribution is the development of a socio-technical framework for auditing the stability of algorithmic systems. This contribution is supplemented with an open-source software library that implements the technical components of the audit, and can be used to conduct similar stability audits of algorithmic systems. We instantiate our framework with the audit of two real-world personality prediction systems, namely Humantic AI and Crystal. The application of our audit framework demonstrates that both these systems show substantial instability with respect to key facets of measurement, and hence cannot be considered valid testing instruments.