# Aligning Characteristic Descriptors with Images for Human-Expert-like Explainability

arXiv: https://arxiv.org/abs/2411.04008 · HTML: https://mlsanity.com/paper/2411.04008/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation](https://mlsanity.com/paper/2308.12604/) (2023)
- [Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanations](https://mlsanity.com/paper/2106.12723/) (2021)
- [Explainable Face Recognition](https://mlsanity.com/paper/2008.00916/) (2020)
- [Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution](https://mlsanity.com/paper/2312.17174/) (2023)
- [How Good is ChatGPT at Face Biometrics? A First Look into Recognition, Soft Biometrics, and Explainability](https://mlsanity.com/paper/2401.13641/) (2024)
- [xCos: An Explainable Cosine Metric for Face Verification Task](https://mlsanity.com/paper/2003.05383/) (2020)
- [Controllable Chest X-Ray Report Generation from Longitudinal Representations](https://mlsanity.com/paper/2310.05881/) (2023)
- [ChatGPT and biometrics: an assessment of face recognition, gender detection, and age estimation capabilities](https://mlsanity.com/paper/2403.02965/) (2024)
- [CLIP-Lung: Textual Knowledge-Guided Lung Nodule Malignancy Prediction](https://mlsanity.com/paper/2304.08013/) (2023)
- [Explainable Model-Agnostic Similarity and Confidence in Face Verification](https://mlsanity.com/paper/2211.13735/) (2022)
- [Efficient Explainable Face Verification based on Similarity Score Argument Backpropagation](https://mlsanity.com/paper/2304.13409/) (2023)
- [I-AI: A Controllable & Interpretable AI System for Decoding Radiologists' Intense Focus for Accurate CXR Diagnoses](https://mlsanity.com/paper/2309.13550/) (2023)
- [Four Principles of Explainable AI as Applied to Biometrics and Facial Forensic Algorithms](https://mlsanity.com/paper/2002.01014/) (2020)

## Abstract (arXiv)

In mission-critical domains such as law enforcement and medical diagnosis, the ability to explain and interpret the outputs of deep learning models is crucial for ensuring user trust and supporting informed decision-making. Despite advancements in explainability, existing methods often fall short in providing explanations that mirror the depth and clarity of those given by human experts. Such expert-level explanations are essential for the dependable application of deep learning models in law enforcement and medical contexts. Additionally, we recognize that most explanations in real-world scenarios are communicated primarily through natural language. Addressing these needs, we propose a novel approach that utilizes characteristic descriptors to explain model decisions by identifying their presence in images, thereby generating expert-like explanations. Our method incorporates a concept bottleneck layer within the model architecture, which calculates the similarity between image and descriptor encodings to deliver inherent and faithful explanations. Through experiments in face recognition and chest X-ray diagnosis, we demonstrate that our approach offers a significant contrast over existing techniques, which are often limited to the use of saliency maps. We believe our approach represents a significant step toward making deep learning systems more accountable, transparent, and trustworthy in the critical domains of face recognition and medical diagnosis.