# Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning

arXiv: https://arxiv.org/abs/1809.10780 · HTML: https://mlsanity.com/paper/1809.10780/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2018.

## Built on by (most cited first, 30 of 56)

- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models](https://mlsanity.com/paper/2404.07983/) (2024)
- [High Fidelity Image Counterfactuals with Probabilistic Causal Models](https://mlsanity.com/paper/2306.15764/) (2023)
- [D'ARTAGNAN: Counterfactual Video Generation](https://mlsanity.com/paper/2206.01651/) (2022)
- [Interventional Sum-Product Networks: Causal Inference with Tractable Probabilistic Models](https://mlsanity.com/paper/2102.10440/) (2021)
- [From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling](https://mlsanity.com/paper/2310.11011/) (2023)
- [Benchmarking Counterfactual Image Generation](https://mlsanity.com/paper/2403.20287/) (2024)
- [Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models](https://mlsanity.com/paper/2404.17735/) (2024)
- [Feature Unlearning for Pre-trained GANs and VAEs](https://mlsanity.com/paper/2303.05699/) (2023)
- [Deep Structural Causal Shape Models](https://mlsanity.com/paper/2208.10950/) (2022)
- [Counterfactual Generative Modeling with Variational Causal Inference](https://mlsanity.com/paper/2410.12730/) (2024)
- [Emerging Synergies in Causality and Deep Generative Models: A Survey](https://mlsanity.com/paper/2301.12351/) (2023)
- [Diffusion Counterfactual Generation with Semantic Abduction](https://mlsanity.com/paper/2506.07883/) (2025)
- [Deep Backtracking Counterfactuals for Causally Compliant Explanations](https://mlsanity.com/paper/2310.07665/) (2023)
- [Modular Learning of Deep Causal Generative Models for High-dimensional Causal Inference](https://mlsanity.com/paper/2401.01426/) (2024)
- [Learning Structural Causal Models through Deep Generative Models: Methods, Guarantees, and Challenges](https://mlsanity.com/paper/2405.05025/) (2024)
- [Turning Federated Learning Systems Into Covert Channels](https://mlsanity.com/paper/2104.10561/) (2021)
- [Semi-Supervised Learning for Deep Causal Generative Models](https://mlsanity.com/paper/2403.18717/) (2024)
- [LeapFactual: Reliable Visual Counterfactual Explanation Using Conditional Flow Matching](https://mlsanity.com/paper/2510.14623/) (2025)
- [Explaining Image Classifiers Using Contrastive Counterfactuals in Generative Latent Spaces](https://mlsanity.com/paper/2206.05257/) (2022)
- [GLANCE: Global to Local Architecture-Neutral Concept-based Explanations](https://mlsanity.com/paper/2207.01917/) (2022)
- [Towards objective and systematic evaluation of bias in artificial intelligence for medical imaging](https://mlsanity.com/paper/2311.02115/) (2023)
- [Causal Generative Explainers using Counterfactual Inference: A Case Study on the Morpho-MNIST Dataset](https://mlsanity.com/paper/2401.11394/) (2024)
- [Conditional Generative Models are Sufficient to Sample from Any Causal Effect Estimand](https://mlsanity.com/paper/2402.07419/) (2024)
- [Common Data Properties Limit Object-Attribute Binding in CLIP](https://mlsanity.com/paper/2507.07985/) (2025)
- [Is More Data All You Need? A Causal Exploration](https://mlsanity.com/paper/2206.02409/) (2022)
- [A Causal Ordering Prior for Unsupervised Representation Learning](https://mlsanity.com/paper/2307.05704/) (2023)
- [Multiple Instance Learning with Auxiliary Task Weighting for Multiple Myeloma Classification](https://mlsanity.com/paper/2107.07805/) (2021)
- [Counterfactual Generation Under Confounding](https://mlsanity.com/paper/2210.12368/) (2022)
- [Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging](https://mlsanity.com/paper/2407.18792/) (2024)

## Abstract (arXiv)

Revealing latent structure in data is an active field of research, having introduced exciting technologies such as variational autoencoders and adversarial networks, and is essential to push machine learning towards unsupervised knowledge discovery. However, a major challenge is the lack of suitable benchmarks for an objective and quantitative evaluation of learned representations. To address this issue we introduce Morpho-MNIST, a framework that aims to answer: "to what extent has my model learned to represent specific factors of variation in the data?" We extend the popular MNIST dataset by adding a morphometric analysis enabling quantitative comparison of trained models, identification of the roles of latent variables, and characterisation of sample diversity. We further propose a set of quantifiable perturbations to assess the performance of unsupervised and supervised methods on challenging tasks such as outlier detection and domain adaptation. Data and code are available at https://github.com/dccastro/Morpho-MNIST.