mlsanityData to 7 Oct 2026

Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data

arXiv 2103.17204 ↗First posted 31 Mar 2021cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

Research in unpaired video translation has mainly focused on short-term temporal consistency by conditioning on neighboring frames. However for transfer from simulated to photorealistic sequences, available information on the underlying geometry offers potential for achieving global consistency across views. We propose a novel approach which combines unpaired image translation with neural rendering to transfer simulated to photorealistic surgical abdominal scenes. By introducing global learnable textures and a lighting-invariant view-consistency loss, our method produces consistent translations of arbitrary views and thus enables long-term consistent video synthesis. We design and test our model to generate video sequences from minimally-invasive surgical abdominal scenes. Because labeled data is often limited in this domain, photorealistic data where ground truth information from the simulated domain is preserved is especially relevant. By extending existing image-based methods to view-consistent videos, we aim to impact the applicability of simulated training and evaluation environments for surgical applications. Code and data: http://opencas.dkfz.de/video-sim2real.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

35 of 54 references.

References · 35

  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  3. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  5. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  6. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  7. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  8. 2018Demystifying MMD GANscited by 1,268
  9. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  10. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  11. 2016Least Squares Generative Adversarial Networkscited by 1,000
  12. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
23 more
  1. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  2. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 637
  3. 2017Unsupervised Image-to-Image Translation Networkscited by 602
  4. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  5. 2020State of the Art on Neural Renderingcited by 276
  6. 2019Deferred Neural Rendering: Image Synthesis using Neural Texturescited by 253
  7. 2019DRIT++: Diverse Image-to-Image Translation via Disentangled Representationscited by 244
  8. 2019Few-Shot Unsupervised Image-to-Image Translationcited by 222
  9. 2019Single Image Portrait Relightingcited by 152
  10. 2017Surgical Data Science: Enabling Next-Generation Surgerycited by 126
  11. 2018Visual Object Networks: Image Generation with Disentangled 3D Representationcited by 124
  12. 2019Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesiscited by 108
  13. 2018Recycle-GAN: Unsupervised Video Retargetingcited by 89
  14. 2020World-Consistent Video-to-Video Synthesiscited by 69
  15. 2019Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translationcited by 38
  16. 2019Mocycle-GAN: Unpaired Video-to-Video Translationcited by 37
  17. 2018MIS-SLAM: Real-time Large Scale Dense Deformable SLAM System in Minimal Invasive Surgery Based on Heterogeneous Computingcited by 29
  18. 2020Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translationcited by 29
  19. 2019Preserving Semantic and Temporal Consistency for Unpaired Video-to-Video Translationcited by 20
  20. 2020Endo-Sim2Real: Consistency learning-based domain adaptation for instrument segmentationcited by 20
  21. 2018Geometric Image Synthesiscited by 16
  22. 2018Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeriescited by 11
  23. 2020Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decompositioncited by 3

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 3; 2024: 9; 2025: 5.

21 papers.

Sort:

By area:

  1. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  2. 2022Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentorscited by 34
  3. 2024VidLPRO: A underlineVideo-underlineLanguage underlinePre-training Framework for underlineRobotic and Laparoscopic Surgerycited by 12
  4. 2023Exploring Semantic Consistency in Unpaired Image Translation to Generate Data for Surgical Applicationscited by 9
  5. 2024Data Augmentation for Surgical Scene Segmentation with Anatomy-Aware Diffusion Modelscited by 8
  6. 2024SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Modelscited by 8
  7. 2024DD-VNB: A Depth-based Dual-Loop Framework for Real-time Visually Navigated Bronchoscopycited by 6
  8. 2022CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopycited by 4
  9. 2022Generalized Product-of-Experts for Learning Multimodal Representations in Noisy Environmentscited by 4
  10. 2024Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgerycited by 4
  11. 2024SurgicaL-CD: Generating Surgical Images via Unpaired Image Translation with Latent Consistency Diffusion Modelscited by 4
  12. 2025SASVi -- Segment Any Surgical Videocited by 4
9 more
  1. 2022Neural Neural Textures Make Sim2Real Consistentcited by 3
  2. 2025Zero-shot Monocular Metric Depth for Endoscopic Imagescited by 2
  3. 2024MeshBrush: Painting the Anatomical Mesh with Neural Stylization for Endoscopycited by 1
  4. 2024Translation-based Video-to-Video Synthesiscited by 1
  5. 2025EndoPBR: Material and Lighting Estimation for Photorealistic Surgical Simulations via Physically-based Renderingcited by 1
  6. 2023RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approachescited by 0
  7. 2024Illumination Histogram Consistency Metric for Quantitative Assessment of Video Sequencescited by 0
  8. 2025Learning from Sparse Point Labels for Dense Carcinosis Localization in Advanced Ovarian Cancer Assessmentcited by 0
  9. 2025Towards Application Aligned Synthetic Surgical Image Synthesiscited by 0