# Cross-Modal Data Programming Enables Rapid Medical Machine Learning

arXiv: https://arxiv.org/abs/1903.11101 · HTML: https://mlsanity.com/paper/1903.11101/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Mar 2019.

## Built on by (most cited first, 30 of 32)

- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [Robust and Efficient Medical Imaging with Self-Supervision](https://mlsanity.com/paper/2205.09723/) (2022)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [End-to-End Weak Supervision](https://mlsanity.com/paper/2107.02233/) (2021)
- [High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning](https://mlsanity.com/paper/2203.01311/) (2022)
- [VisualCheXbert: Addressing the Discrepancy Between Radiology Report Labels and Image Labels](https://mlsanity.com/paper/2102.11467/) (2021)
- [Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models](https://mlsanity.com/paper/2311.00287/) (2023)
- [Learning Hyper Label Model for Programmatic Weak Supervision](https://mlsanity.com/paper/2207.13545/) (2022)
- [Creating Training Sets via Weak Indirect Supervision](https://mlsanity.com/paper/2110.03484/) (2021)
- [Active WeaSuL: Improving Weak Supervision with Active Learning](https://mlsanity.com/paper/2104.14847/) (2021)
- [Lifting Weak Supervision To Structured Prediction](https://mlsanity.com/paper/2211.13375/) (2022)
- [Weak Supervision for Affordable Modeling of Electrocardiogram Data](https://mlsanity.com/paper/2201.02936/) (2022)
- [A Systematic Review of Generalization Research in Medical Image Classification](https://mlsanity.com/paper/2403.12167/) (2024)
- [AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labels](https://mlsanity.com/paper/2208.14362/) (2022)
- [Exploring Semi-Supervised Learning for Predicting Listener Backchannels](https://mlsanity.com/paper/2101.01899/) (2021)
- [Automatic Synthesis of Diverse Weak Supervision Sources for Behavior Analysis](https://mlsanity.com/paper/2111.15186/) (2021)
- [Towards trustworthy seizure onset detection using workflow notes](https://mlsanity.com/paper/2306.08728/) (2023)
- [Challenges for cognitive decoding using deep learning methods](https://mlsanity.com/paper/2108.06896/) (2021)
- [Binary Classification with Positive Labeling Sources](https://mlsanity.com/paper/2208.01704/) (2022)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [Automating Weak Label Generation for Data Programming with Clinicians in the Loop](https://mlsanity.com/paper/2407.07982/) (2024)
- [Generative Modeling Helps Weak Supervision (and Vice Versa)](https://mlsanity.com/paper/2203.12023/) (2022)
- [Integrated Weak Learning](https://mlsanity.com/paper/2206.09496/) (2022)
- [Regularized Data Programming with Automated Bayesian Prior Selection](https://mlsanity.com/paper/2210.08677/) (2022)
- [MGA: Medical generalist agent through text-guided knowledge transformation](https://mlsanity.com/paper/2303.08562/) (2023)
- [A Benchmark Generative Probabilistic Model for Weak Supervised Learning](https://mlsanity.com/paper/2303.17841/) (2023)

## Abstract (arXiv)

Labeling training datasets has become a key barrier to building medical machine learning models. One strategy is to generate training labels programmatically, for example by applying natural language processing pipelines to text reports associated with imaging studies. We propose cross-modal data programming, which generalizes this intuitive strategy in a theoretically-grounded way that enables simpler, clinician-driven input, reduces required labeling time, and improves with additional unlabeled data. In this approach, clinicians generate training labels for models defined over a target modality (e.g. images or time series) by writing rules over an auxiliary modality (e.g. text reports). The resulting technical challenge consists of estimating the accuracies and correlations of these rules; we extend a recent unsupervised generative modeling technique to handle this cross-modal setting in a provably consistent way. Across four applications in radiography, computed tomography, and electroencephalography, and using only several hours of clinician time, our approach matches or exceeds the efficacy of physician-months of hand-labeling with statistical significance, demonstrating a fundamentally faster and more flexible way of building machine learning models in medicine.