# Reproducibility in Multiple Instance Learning: A Case For Algorithmic Unit Tests

arXiv: https://arxiv.org/abs/2310.17867 · HTML: https://mlsanity.com/paper/2310.17867/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Are we done with ImageNet?](https://mlsanity.com/paper/2006.07159/) (2020)
- [Accounting for Variance in Machine Learning Benchmarks](https://mlsanity.com/paper/2103.03098/) (2021)
- [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019)
- [Multiple Instance Learning: A Survey of Problem Characteristics and Applications](https://mlsanity.com/paper/1612.03365/) (2016)
- [Revisiting Multiple Instance Neural Networks](https://mlsanity.com/paper/1610.02501/) (2016)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Do We Train on Test Data? Purging CIFAR of Near-Duplicates](https://mlsanity.com/paper/1902.00423/) (2019)
- [Modern Hopfield Networks and Attention for Immune Repertoire Classification](https://mlsanity.com/paper/2007.13505/) (2020)
- [Differentiable Zooming for Multiple Instance Learning on Whole-Slide Images](https://mlsanity.com/paper/2204.12454/) (2022)
- [Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets](https://mlsanity.com/paper/1709.09500/) (2017)
- [Multiple instance learning with graph neural networks](https://mlsanity.com/paper/1906.04881/) (2019)
- ["Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?](https://mlsanity.com/paper/2107.02278/) (2021)
- [Reproducibility in Optimization: Theoretical Framework and Limits](https://mlsanity.com/paper/2202.04598/) (2022)
- [Non-Negative Networks Against Adversarial Attacks](https://mlsanity.com/paper/1806.06108/) (2018)
- [Hyperparameter Optimization Is Deceiving Us, and How to Stop It](https://mlsanity.com/paper/2102.03034/) (2021)
- [Multi-Instance Causal Representation Learning for Instance Label Prediction and Out-of-Distribution Generalization](https://mlsanity.com/paper/2202.12570/) (2022)
- [Bag Graph: Multiple Instance Learning using Bayesian Graph Neural Networks](https://mlsanity.com/paper/2202.11132/) (2022)
- [Nested Multiple Instance Learning in Modelling of HTTP network traffic](https://mlsanity.com/paper/2002.04059/) (2020)
- [A Coreset Learning Reality Check](https://mlsanity.com/paper/2301.06163/) (2023)
- [Multiple Instance Learning for Malware Classification](https://mlsanity.com/paper/1705.02268/) (2017)
- [mil-benchmarks: Standardized Evaluation of Deep Multiple-Instance Learning Techniques](https://mlsanity.com/paper/2105.01443/) (2021)

## Built on by (most cited first, 11 of 11)

- [What Do Machine Learning Researchers Mean by "Reproducible"?](https://mlsanity.com/paper/2412.03854/) (2024)
- [From Correlation to Causation: Max-Pooling-Based Multi-Instance Learning Leads to More Robust Whole Slide Image Classification](https://mlsanity.com/paper/2408.09449/) (2024)
- [Cracking Instance Jigsaw Puzzles: An Alternative to Multiple Instance Learning for Whole Slide Image Analysis](https://mlsanity.com/paper/2507.08178/) (2025)
- [torchmil: A PyTorch-based library for deep Multiple Instance Learning](https://mlsanity.com/paper/2509.08129/) (2025)
- [Synthetic Data Reveals Generalization Gaps in Correlated Multiple Instance Learning](https://mlsanity.com/paper/2510.25759/) (2025)
- [A Multi-Dataset Benchmark of Multiple Instance Learning for 3D Neuroimage Classification](https://mlsanity.com/paper/2604.26807/) (2026)
- [A Vector Symbolic Approach to Multiple Instance Learning](https://mlsanity.com/paper/2511.16795/) (2025)
- [Bayesian Meta-Analyses Could Be More: A Case Study in Trial of Labor After a Cesarean-section Outcomes and Complications](https://mlsanity.com/paper/2601.10089/) (2026)
- [Normal Guidance is what Attention Needs](https://mlsanity.com/paper/2605.27306/) (2026)
- [Weakly-supervised Kidney Tumor Classification from CT Scans with Multi-Instance Learning and Anatomical Filtering](https://mlsanity.com/paper/2609.07178/) (2026)
- [Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets](https://mlsanity.com/paper/2609.23995/) (2026)

## Abstract (arXiv)

Multiple Instance Learning (MIL) is a sub-domain of classification problems with positive and negative labels and a "bag" of inputs, where the label is positive if and only if a positive element is contained within the bag, and otherwise is negative. Training in this context requires associating the bag-wide label to instance-level information, and implicitly contains a causal assumption and asymmetry to the task (i.e., you can't swap the labels without changing the semantics). MIL problems occur in healthcare (one malignant cell indicates cancer), cyber security (one malicious executable makes an infected computer), and many other tasks. In this work, we examine five of the most prominent deep-MIL models and find that none of them respects the standard MIL assumption. They are able to learn anti-correlated instances, i.e., defaulting to "positive" labels until seeing a negative counter-example, which should not be possible for a correct MIL model. We suspect that enhancements and other works derived from these models will share the same issue. In any context in which these models are being used, this creates the potential for learning incorrect models, which creates risk of operational failure. We identify and demonstrate this problem via a proposed "algorithmic unit test", where we create synthetic datasets that can be solved by a MIL respecting model, and which clearly reveal learning that violates MIL assumptions. The five evaluated methods each fail one or more of these tests. This provides a model-agnostic way to identify violations of modeling assumptions, which we hope will be useful for future development and evaluation of MIL models.