# Human-AI Collaborative Multi-modal Multi-rater Learning for Endometriosis Diagnosis

arXiv: https://arxiv.org/abs/2409.02046 · HTML: https://mlsanity.com/paper/2409.02046/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 64 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 30 of 30)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1808.00948/) (2018)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019)
- [Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty](https://mlsanity.com/paper/2006.06015/) (2020)
- [Trusted Multi-View Classification](https://mlsanity.com/paper/2102.02051/) (2021)
- [Unpaired Multi-modal Segmentation via Knowledge Distillation](https://mlsanity.com/paper/2001.03111/) (2020)
- [Who Said What: Modeling Individual Labelers Improves Classification](https://mlsanity.com/paper/1703.08774/) (2017)
- [Differentiable Learning Under Triage](https://mlsanity.com/paper/2103.08902/) (2021)
- [ProMix: Combating Label Noise via Maximizing Clean Sample Utility](https://mlsanity.com/paper/2207.10276/) (2022)
- [Learning Disentangled Representations in the Imaging Domain](https://mlsanity.com/paper/2108.12043/) (2021)
- [Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality](https://mlsanity.com/paper/2310.01035/) (2023)
- [Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration](https://mlsanity.com/paper/2109.14591/) (2021)
- [Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts](https://mlsanity.com/paper/2206.07948/) (2022)
- [Do Humans Trust Advice More if it Comes from AI? An Analysis of Human-AI Interactions](https://mlsanity.com/paper/2107.07015/) (2021)
- [Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning](https://mlsanity.com/paper/2103.10211/) (2021)
- [Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possible](https://mlsanity.com/paper/2101.05360/) (2021)
- [CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators](https://mlsanity.com/paper/2210.06812/) (2022)
- [SSR: An Efficient and Robust Framework for Learning with Unknown Label Noise](https://mlsanity.com/paper/2111.11288/) (2021)
- [Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Prediction](https://mlsanity.com/paper/2207.10851/) (2022)
- [Human and AI Perceptual Differences in Image Classification Errors](https://mlsanity.com/paper/2304.08733/) (2023)
- [Distilling Missing Modality Knowledge from Ultrasound for Endometriosis Diagnosis with Magnetic Resonance Images](https://mlsanity.com/paper/2307.02000/) (2023)

## Built on by (most cited first, 1 of 1)

- [Cross-Modal MRI Ovary Segmentation in Endometriosis Using Unpaired TVUS Prototype Priors](https://mlsanity.com/paper/2608.18515/) (2026)

## Abstract (arXiv)

Endometriosis, affecting about 10% of individuals assigned female at birth, is challenging to diagnose and manage. Diagnosis typically involves the identification of various signs of the disease using either laparoscopic surgery or the analysis of T1/T2 MRI images, with the latter being quicker and cheaper but less accurate. A key diagnostic sign of endometriosis is the obliteration of the Pouch of Douglas (POD). However, even experienced clinicians struggle with accurately classifying POD obliteration from MRI images, which complicates the training of reliable AI models. In this paper, we introduce the Human-AI Collaborative Multi-modal Multi-rater Learning (HAICOMM) methodology to address the challenge above. HAICOMM is the first method that explores three important aspects of this problem: 1) multi-rater learning to extract a cleaner label from the multiple "noisy" labels available per training sample; 2) multi-modal learning to leverage the presence of T1/T2 MRI images for training and testing; and 3) human-AI collaboration to build a system that leverages the predictions from clinicians and the AI model to provide more accurate classification than standalone clinicians and AI models. Presenting results on the multi-rater T1/T2 MRI endometriosis dataset that we collected to validate our methodology, the proposed HAICOMM model outperforms an ensemble of clinicians, noisy-label learning models, and multi-rater learning methods.