# MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video

arXiv: https://arxiv.org/abs/2408.03761 · HTML: https://mlsanity.com/paper/2408.03761/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models](https://mlsanity.com/paper/2302.07257/) (2023)
- [SegVol: Universal and Interactive Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2311.13385/) (2023)
- [Video Summarization Using Deep Neural Networks: A Survey](https://mlsanity.com/paper/2101.06072/) (2021)
- [Hierarchical Video-Moment Retrieval and Step-Captioning](https://mlsanity.com/paper/2303.16406/) (2023)
- [Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignment](https://mlsanity.com/paper/2112.15011/) (2021)
- [SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound](https://mlsanity.com/paper/1612.05601/) (2016)
- [StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos](https://mlsanity.com/paper/2304.13265/) (2023)
- [Progressive Video Summarization via Multimodal Self-supervised Learning](https://mlsanity.com/paper/2201.02494/) (2022)
- [IntentVizor: Towards Generic Query Guided Interactive Video Summarization](https://mlsanity.com/paper/2109.14834/) (2021)
- [AutoFB: Automating Fetal Biometry Estimation from Standard Ultrasound Planes](https://mlsanity.com/paper/2107.05255/) (2021)
- [Ultrasound Video Summarization using Deep Reinforcement Learning](https://mlsanity.com/paper/2005.09531/) (2020)
- [Experience and Evidence are the eyes of an excellent summarizer! Towards Knowledge Infused Multi-modal Clinical Conversation Summarization](https://mlsanity.com/paper/2309.15739/) (2023)
- [Breast Ultrasound Report Generation using LangChain](https://mlsanity.com/paper/2312.03013/) (2023)
- [AI assisted method for efficiently generating breast ultrasound screening reports](https://mlsanity.com/paper/2107.13431/) (2021)

## Built on by (most cited first, 9 of 9)

- [U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding](https://mlsanity.com/paper/2505.17779/) (2025)
- [TPA: Temporal Prompt Alignment for Fetal Congenital Heart Defect Classification](https://mlsanity.com/paper/2508.15298/) (2025)
- [Beyond Benchmarks of IUGC: Rethinking Requirements of Deep Learning Methods for Intrapartum Ultrasound Biometry from Fetal Ultrasound Videos](https://mlsanity.com/paper/2602.12922/) (2026)
- [Fusing Radiomic Features with Deep Representations for Gestational Age Estimation in Fetal Ultrasound Images](https://mlsanity.com/paper/2506.20407/) (2025)
- [Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenization](https://mlsanity.com/paper/2509.05695/) (2025)
- [FetalAgents: A Multi-Agent System for Fetal Ultrasound Image and Video Analysis](https://mlsanity.com/paper/2603.09733/) (2026)
- [Structure-Augmented Standard Plane Detection with Temporal Aggregation in Blind-Sweep Fetal Ultrasound](https://mlsanity.com/paper/2604.20591/) (2026)
- [A Clinician-Centered Pipeline for Annotation and Evaluation in Ultrasound AI Studies](https://mlsanity.com/paper/2606.19174/) (2026)
- [FetSelect: Task-Specific Architectures and Self-Supervised Learning for Automated Fetal Ultrasound Frame Selection](https://mlsanity.com/paper/2606.22487/) (2026)

## Abstract (arXiv)

We present the first automated multimodal summary generation system, MMSummary, for medical imaging video, particularly with a focus on fetal ultrasound analysis. Imitating the examination process performed by a human sonographer, MMSummary is designed as a three-stage pipeline, progressing from keyframe detection to keyframe captioning and finally anatomy segmentation and measurement. In the keyframe detection stage, an innovative automated workflow is proposed to progressively select a concise set of keyframes, preserving sufficient video information without redundancy. Subsequently, we adapt a large language model to generate meaningful captions for fetal ultrasound keyframes in the keyframe captioning stage. If a keyframe is captioned as fetal biometry, the segmentation and measurement stage estimates biometric parameters by segmenting the region of interest according to the textual prior. The MMSummary system provides comprehensive summaries for fetal ultrasound examinations and based on reported experiments is estimated to reduce scanning time by approximately 31.5%, thereby suggesting the potential to enhance clinical workflow efficiency.