# Representation Fidelity:Auditing Algorithmic Decisions About Humans Using Self-Descriptions

arXiv: https://arxiv.org/abs/2603.05136 · HTML: https://mlsanity.com/paper/2603.05136/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 59 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle](https://mlsanity.com/paper/1901.10002/) (2019)
- [MotionBERT: A Unified Perspective on Learning Human Motion Representations](https://mlsanity.com/paper/2210.06551/) (2022)
- [TEACH: Temporal Action Composition for 3D Humans](https://mlsanity.com/paper/2209.04066/) (2022)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Outcome Indistinguishability](https://mlsanity.com/paper/2011.13426/) (2020)
- [Online Learning with an Unknown Fairness Metric](https://mlsanity.com/paper/1802.06936/) (2018)
- [Fairness Through Computationally-Bounded Awareness](https://mlsanity.com/paper/1803.03239/) (2018)
- [Human Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction](https://mlsanity.com/paper/1802.09548/) (2018)
- [Difficult Lessons on Social Prediction from Wisconsin Public Schools](https://mlsanity.com/paper/2304.06205/) (2023)
- [Probably Approximately Metric-Fair Learning](https://mlsanity.com/paper/1803.03242/) (2018)
- [Taxonomizing and Measuring Representational Harms: A Look at Image Tagging](https://mlsanity.com/paper/2305.01776/) (2023)
- [Subverting machines, fluctuating identities: Re-learning human categorization](https://mlsanity.com/paper/2205.13740/) (2022)
- [Taxonomizing Representational Harms using Speech Act Theory](https://mlsanity.com/paper/2504.00928/) (2025)
- [Embedding Atlas: Low-Friction, Interactive Embedding Visualization](https://mlsanity.com/paper/2505.06386/) (2025)
- [Bias Begins with Data: The FairGround Corpus for Robust and Reproducible Research on Algorithmic Fairness](https://mlsanity.com/paper/2510.22363/) (2025)

## Abstract (arXiv)

This paper introduces a new dimension for validating algorithmic decisions about humans by measuring the fidelity of their representations. Representation Fidelity measures if decisions about a person rest on reasonable grounds. We propose to operationalize this notion by measuring the distance between two representations of the same person: (1) an externally prescribed input representation on which the decision is based, and (2) a self-description provided by the human subject of the decision, used solely to validate the input representation. We examine the nature of discrepancies between these representations, how such discrepancies can be quantified, and derive a generic typology of representation mismatches that determine the degree of representation fidelity. We further present the first benchmark for evaluating representation fidelity based on a dataset of loan-granting decisions. Our Loan-Granting Self-Representations Corpus 2025 consists of a large corpus of 30 000 synthetic natural language self-descriptions derived from corresponding representations of applicants in the German Credit Dataset, along with expert annotations of representation mismatches between each pair of representations.