# MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery

arXiv: https://arxiv.org/abs/2602.12407 · HTML: https://mlsanity.com/paper/2602.12407/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 59 references that are in the map.
- First posted on arXiv 12 Feb 2026.

## Builds on (most cited first, 18 of 18)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [MIcro-Surgical Anastomose Workflow recognition challenge report](https://mlsanity.com/paper/2103.13111/) (2021)
- [Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery](https://mlsanity.com/paper/1806.05796/) (2018)
- [Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery](https://mlsanity.com/paper/2212.01683/) (2022)
- [Real-Time Context-aware Detection of Unsafe Events in Robot-Assisted Surgery](https://mlsanity.com/paper/2005.03611/) (2020)
- [Multimodal Transformers for Real-Time Surgical Activity Prediction](https://mlsanity.com/paper/2403.06705/) (2024)
- [Analysis of Executional and Procedural Errors in Dry-lab Robotic Surgery Experiments](https://mlsanity.com/paper/2106.11962/) (2021)
- [DESK: A Robotic Activity Dataset for Dexterous Surgical Skills Transfer to Medical Robots](https://mlsanity.com/paper/1903.00959/) (2019)
- [COMPASS: A Formal Framework and Aggregate Dataset for Generalized Surgical Procedure Modeling](https://mlsanity.com/paper/2209.06424/) (2022)
- [Comprehensive Robotic Cholecystectomy Dataset (CRCD): Integrating Kinematics, Pedal Signals, and Endoscopic Videos](https://mlsanity.com/paper/2312.01183/) (2023)

## Built on by (most cited first, 1 of 1)

- [Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery](https://mlsanity.com/paper/2606.23593/) (2026)

## Abstract (arXiv)

Background: Robot-assisted minimally invasive surgery (RMIS) research increasingly relies on multimodal data, yet access to proprietary robot telemetry remains a major barrier. We introduce MiDAS, an open-source, platform-agnostic system enabling time-synchronized, non-invasive multimodal data acquisition across surgical robotic platforms. Methods: MiDAS integrates electromagnetic and RGB-D hand tracking, foot pedal sensing, and surgical video capturing without requiring proprietary robot interfaces. We validated MiDAS on the open-source Raven-II and the clinical da Vinci Xi by collecting multimodal datasets of peg transfer and hernia repair suturing tasks performed by surgical residents. Correlation analysis and downstream gesture recognition experiments were conducted. Results: External hand and foot sensing closely approximated internal robot kinematics and non-invasive motion signals achieved gesture recognition performance comparable to proprietary telemetry. Conclusion: MiDAS enables reproducible multimodal RMIS data collection and is released with annotated datasets, including the first multimodal dataset capturing hernia repair suturing on high-fidelity simulation models.