# Multimodal Learning for Embryo Viability Prediction in Clinical IVF

arXiv: https://arxiv.org/abs/2410.15581 · HTML: https://mlsanity.com/paper/2410.15581/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [Learning Vector Quantized Shape Code for Amodal Blastomere Instance Segmentation](https://mlsanity.com/paper/2012.00985/) (2020)
- [Automated Measurements of Key Morphological Features of Human Embryos for IVF](https://mlsanity.com/paper/2006.00067/) (2020)
- [Developmental Stage Classification of Embryos Using Two-Stream Neural Network with Linear-Chain Conditional Random Field](https://mlsanity.com/paper/2107.06360/) (2021)
- [Embryo staging with weakly-supervised region selection and dynamically-decoded predictions](https://mlsanity.com/paper/1904.04419/) (2019)

## Built on by (most cited first, 4 of 4)

- [Multi-Modal Artificial Intelligence of Embryo Grading and Pregnancy Prediction in Assisted Reproductive Technology: A Review](https://mlsanity.com/paper/2505.20306/) (2025)
- [Time-Lapse Video-Based Embryo Grading via Complementary Spatial-Temporal Pattern Mining](https://mlsanity.com/paper/2506.04950/) (2025)
- [Cytoplasmic Strings Analysis in Human Embryo Time-Lapse Videos using Deep Learning Framework](https://mlsanity.com/paper/2512.09461/) (2025)
- [Predicting Blastocyst Formation in IVF: Integrating DINOv2 and Attention-Based LSTM on Time-Lapse Embryo Images](https://mlsanity.com/paper/2604.16505/) (2026)

## Abstract (arXiv)

In clinical In-Vitro Fertilization (IVF), identifying the most viable embryo for transfer is important to increasing the likelihood of a successful pregnancy. Traditionally, this process involves embryologists manually assessing embryos' static morphological features at specific intervals using light microscopy. This manual evaluation is not only time-intensive and costly, due to the need for expert analysis, but also inherently subjective, leading to variability in the selection process. To address these challenges, we develop a multimodal model that leverages both time-lapse video data and Electronic Health Records (EHRs) to predict embryo viability. One of the primary challenges of our research is to effectively combine time-lapse video and EHR data, owing to their inherent differences in modality. We comprehensively analyze our multimodal model with various modality inputs and integration approaches. Our approach will enable fast and automated embryo viability predictions in scale for clinical IVF.