# Multimodal Transformers for Real-Time Surgical Activity Prediction

arXiv: https://arxiv.org/abs/2403.06705 · HTML: https://mlsanity.com/paper/2403.06705/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2024.

## Built on by (most cited first, 11 of 11)

- [SutureFormer: Learning Surgical Trajectories via Goal-conditioned Offline RL in Pixel Space](https://mlsanity.com/paper/2603.26720/) (2026)
- [Multi-Modal Gesture Recognition from Video and Surgical Tool Pose Information via Motion Invariants](https://mlsanity.com/paper/2503.15647/) (2025)
- [M2R2: MultiModal Robotic Representation for Temporal Action Segmentation](https://mlsanity.com/paper/2504.18662/) (2025)
- [DARIL: When Imitation Learning outperforms Reinforcement Learning in Surgical Action Planning](https://mlsanity.com/paper/2507.05011/) (2025)
- [MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery](https://mlsanity.com/paper/2602.12407/) (2026)
- [EgoEMS: A High-Fidelity Multimodal Egocentric Dataset for Cognitive Assistance in Emergency Medical Services](https://mlsanity.com/paper/2511.09894/) (2025)
- [Towards Comprehensive Real-Time Scene Understanding in Ophthalmic Surgery through Multimodal Image Fusion](https://mlsanity.com/paper/2603.25555/) (2026)
- [SHANDS: A Multi-View Dataset and Benchmark for Surgical Hand-Gesture and Error Recognition Toward Medical Training](https://mlsanity.com/paper/2603.26400/) (2026)
- [On The Application of Linear Attention in Multimodal Transformers](https://mlsanity.com/paper/2604.10064/) (2026)
- [Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery](https://mlsanity.com/paper/2606.23593/) (2026)
- [Multi-modal Representation Fusion for Long-term Microsurgical Anastomosis Trajectory Anticipation](https://mlsanity.com/paper/2610.14492/) (2026)

## Abstract (arXiv)

Real-time recognition and prediction of surgical activities are fundamental to advancing safety and autonomy in robot-assisted surgery. This paper presents a multimodal transformer architecture for real-time recognition and prediction of surgical gestures and trajectories based on short segments of kinematic and video data. We conduct an ablation study to evaluate the impact of fusing different input modalities and their representations on gesture recognition and prediction performance. We perform an end-to-end assessment of the proposed architecture using the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) dataset. Our model outperforms the state-of-the-art (SOTA) with 89.5% accuracy for gesture prediction through effective fusion of kinematic features with spatial and contextual video features. It achieves the real-time performance of 1.1-1.3ms for processing a 1-second input window by relying on a computationally efficient model.