# Pooling Image Datasets With Multiple Covariate Shift and Imbalance

arXiv: https://arxiv.org/abs/2403.02598 · HTML: https://mlsanity.com/paper/2403.02598/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 84 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Unified Deep Supervised Domain Adaptation and Generalization](https://mlsanity.com/paper/1709.10190/) (2017)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [From Variational to Deterministic Autoencoders](https://mlsanity.com/paper/1903.12436/) (2019)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [Topological Autoencoders](https://mlsanity.com/paper/1906.00722/) (2019)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [Invariant Representations without Adversarial Training](https://mlsanity.com/paper/1805.09458/) (2018)
- [Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network](https://mlsanity.com/paper/1802.09655/) (2018)
- [A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation](https://mlsanity.com/paper/2010.14766/) (2020)
- [Learning Disentangled Representations in the Imaging Domain](https://mlsanity.com/paper/2108.12043/) (2021)
- [Backprop as Functor: A compositional perspective on supervised learning](https://mlsanity.com/paper/1711.10455/) (2017)
- [Categorical Foundations of Gradient-Based Learning](https://mlsanity.com/paper/2103.01931/) (2021)
- [Category Theory in Machine Learning](https://mlsanity.com/paper/2106.07032/) (2021)
- [Quantum Permutation Synchronization](https://mlsanity.com/paper/2101.07755/) (2021)
- [Scanner Invariant Representations for Diffusion MRI Harmonization](https://mlsanity.com/paper/1904.05375/) (2019)
- [Probabilistic Permutation Synchronization using the Riemannian Structure of the Birkhoff Polytope](https://mlsanity.com/paper/1904.05814/) (2019)
- [Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization](https://mlsanity.com/paper/2102.06315/) (2021)
- [Fairness and Accuracy under Domain Generalization](https://mlsanity.com/paper/2301.13323/) (2023)
- [FairALM: Augmented Lagrangian Method for Training Fair Models with Little Regret](https://mlsanity.com/paper/2004.01355/) (2020)
- [Reconstruction-Based Disentanglement for Pose-invariant Face Recognition](https://mlsanity.com/paper/1702.03041/) (2017)
- [Homomorphism Autoencoder -- Learning Group Structured Representations from Observed Transitions](https://mlsanity.com/paper/2207.12067/) (2022)
- [Lenses and Learners](https://mlsanity.com/paper/1903.03671/) (2019)

## Built on by (most cited first, 4 of 4)

- [ManifoldGD: Training-Free Hierarchical Manifold Guidance for Diffusion-Based Dataset Distillation](https://mlsanity.com/paper/2602.23295/) (2026)
- [Is Exchangeability better than I.I.D to handle Data Distribution Shifts while Pooling Data for Data-scarce Medical image segmentation?](https://mlsanity.com/paper/2507.19575/) (2025)
- [Structured Credal Learning](https://mlsanity.com/paper/2603.14070/) (2026)
- [MAGNET: Manifold-Aware Graph Diffusion Network for Connectome Generation](https://mlsanity.com/paper/2610.17014/) (2026)

## Abstract (arXiv)

Small sample sizes are common in many disciplines, which necessitates pooling roughly similar datasets across multiple institutions to study weak but relevant associations between images and disease outcomes. Such data often manifest shift/imbalance in covariates (i.e., secondary non-imaging data). Controlling for such nuisance variables is common within standard statistical analysis, but the ideas do not directly apply to overparameterized models. Consequently, recent work has shown how strategies from invariant representation learning provides a meaningful starting point, but the current repertoire of methods is limited to accounting for shifts/imbalances in just a couple of covariates at a time. In this paper, we show how viewing this problem from the perspective of Category theory provides a simple and effective solution that completely avoids elaborate multi-stage training pipelines that would otherwise be needed. We show the effectiveness of this approach via extensive experiments on real datasets. Further, we discuss how this style of formulation offers a unified perspective on at least 5+ distinct problem settings, from self-supervised learning to matching problems in 3D reconstruction.