# Future Frame Prediction for Robot-assisted Surgery

arXiv: https://arxiv.org/abs/2103.10308 · HTML: https://mlsanity.com/paper/2103.10308/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 18 Mar 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Action-Conditional Video Prediction using Deep Networks in Atari Games](https://mlsanity.com/paper/1507.08750/) (2015)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [Unsupervised Learning of Disentangled Representations from Video](https://mlsanity.com/paper/1705.10915/) (2017)
- [Video Pixel Networks](https://mlsanity.com/paper/1610.00527/) (2016)
- [Unsupervised Learning of Object Structure and Dynamics from Videos](https://mlsanity.com/paper/1906.07889/) (2019)
- [Learning to Generate Long-term Future via Hierarchical Prediction](https://mlsanity.com/paper/1704.05831/) (2017)
- [High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks](https://mlsanity.com/paper/1911.01655/) (2019)
- [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019)
- [VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation](https://mlsanity.com/paper/1903.01434/) (2019)
- [Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video](https://mlsanity.com/paper/1907.07899/) (2019)
- [Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction](https://mlsanity.com/paper/2002.09905/) (2020)
- [Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video](https://mlsanity.com/paper/1907.11454/) (2019)
- [Unsupervised Keypoint Learning for Guiding Class-Conditional Video Prediction](https://mlsanity.com/paper/1910.02027/) (2019)
- [Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Search](https://mlsanity.com/paper/2002.08718/) (2020)
- [Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning](https://mlsanity.com/paper/2007.11319/) (2020)
- [Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification](https://mlsanity.com/paper/1806.08089/) (2018)
- [CFCM: Segmentation via Coarse to Fine Context Memory](https://mlsanity.com/paper/1806.01413/) (2018)
- [Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos](https://mlsanity.com/paper/2006.00545/) (2020)

## Built on by (most cited first, 11 of 11)

- [TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2202.08453/) (2022)
- [One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video](https://mlsanity.com/paper/2103.12988/) (2021)
- [Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery](https://mlsanity.com/paper/2212.01683/) (2022)
- [Rethinking Boundary Detection in Deep Learning Models for Medical Image Segmentation](https://mlsanity.com/paper/2305.00678/) (2023)
- [Rethinking Boundary Detection in Deep Learning-Based Medical Image Segmentation](https://mlsanity.com/paper/2505.04652/) (2025)
- [Generalized Task-Driven Medical Image Quality Enhancement with Gradient Promotion](https://mlsanity.com/paper/2501.01114/) (2025)
- [Anti-aliasing Predictive Coding Network for Future Video Frame Prediction](https://mlsanity.com/paper/2301.05421/) (2023)
- [Video-Instrument Synergistic Network for Referring Video Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2308.09475/) (2023)
- [SFGANS Self-supervised Future Generator for human ActioN Segmentation](https://mlsanity.com/paper/2401.00438/) (2023)
- [Fuzzy Attention-based Border Rendering Network for Lung Organ Segmentation](https://mlsanity.com/paper/2406.16189/) (2024)
- [Surg-UniWorld: A Unified Surgical World Model with Multimodal Control Experts](https://mlsanity.com/paper/2608.06770/) (2026)

## Abstract (arXiv)

Predicting future frames for robotic surgical video is an interesting, important yet extremely challenging problem, given that the operative tasks may have complex dynamics. Existing approaches on future prediction of natural videos were based on either deterministic models or stochastic models, including deep recurrent neural networks, optical flow, and latent space modeling. However, the potential in predicting meaningful movements of robots with dual arms in surgical scenarios has not been tapped so far, which is typically more challenging than forecasting independent motions of one arm robots in natural scenarios. In this paper, we propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools. Furthermore, we add the invariant prior information from the gesture class into the generation process to constrain the latent space of our model. To our best knowledge, this is the first time that the future frames of dual arm robots are predicted considering their unique characteristics relative to general robotic videos. Experiments demonstrate that our model gains more stable and realistic future frame prediction scenes with the suturing task on the public JIGSAWS dataset.