# An objective comparison of methods for augmented reality in laparoscopic liver resection by preoperative-to-intraoperative image fusion arXiv: https://arxiv.org/abs/2401.15753 ยท HTML: https://mlsanity.com/paper/2401.15753/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 48 references that are in the map. - First posted on arXiv 28 Jan 2024. ## Builds on (most cited first, 22 of 22) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016) - [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017) - [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021) - [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015) - [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017) - [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017) - [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018) - [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018) - [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015) - [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020) - [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020) - [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019) - [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016) - [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017) - [MeshCNN: A Network with an Edge](https://mlsanity.com/paper/1809.05910/) (2018) - [Methods and open-source toolkit for analyzing and visualizing challenge results](https://mlsanity.com/paper/1910.05121/) (2019) - [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021) - [Towards a Guideline for Evaluation Metrics in Medical Image Segmentation](https://mlsanity.com/paper/2202.05273/) (2022) - [Intraoperative Liver Surface Completion with Graph Convolutional VAE](https://mlsanity.com/paper/2009.03871/) (2020) ## Built on by (most cited first, 16 of 16) - [Depth-Driven Geometric Prompt Learning for Laparoscopic Liver Landmark Detection](https://mlsanity.com/paper/2406.17858/) (2024) - [Landmark-Free Preoperative-to-Intraoperative Registration in Laparoscopic Liver Resection](https://mlsanity.com/paper/2504.15152/) (2025) - [Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoning](https://mlsanity.com/paper/2409.07238/) (2024) - [A Review on Organ Deformation Modeling Approaches for Reliable Surgical Navigation using Augmented Reality](https://mlsanity.com/paper/2408.02713/) (2024) - [Synergistic Bleeding Region and Point Detection in Laparoscopic Surgical Videos](https://mlsanity.com/paper/2503.22174/) (2025) - [BCRNet: Enhancing Landmark Detection in Laparoscopic Liver Surgery via Bezier Curve Refinement](https://mlsanity.com/paper/2506.15279/) (2025) - [Topology-Constrained Learning for Efficient Laparoscopic Liver Landmark Detection](https://mlsanity.com/paper/2507.00519/) (2025) - [HRVVS: A High-resolution Video Vasculature Segmentation Network via Hierarchical Autoregressive Residual Priors](https://mlsanity.com/paper/2507.22530/) (2025) - [CECT-Mamba: a Hierarchical Contrast-enhanced-aware Model for Pancreatic Tumor Subtyping from Multi-phase CECT](https://mlsanity.com/paper/2509.12777/) (2025) - [Subsampled Randomized Fourier GaLore for Adapting Foundation Models in Depth-Driven Liver Landmark Segmentation](https://mlsanity.com/paper/2511.03163/) (2025) - [Depth Augmented and FE Free 3D/2D Liver Registration for Laparoscopic Liver AR](https://mlsanity.com/paper/2602.17517/) (2026) - [PIVOTS: Aligning unseen Structures using Preoperative to Intraoperative Volume-To-Surface Registration for Liver Navigation](https://mlsanity.com/paper/2507.20337/) (2025) - [Preoperative-to-intraoperative Liver Registration for Laparoscopic Surgery via Latent-Grounded Correspondence Constraints](https://mlsanity.com/paper/2603.01720/) (2026) - [Warm-Started Reinforcement Learning for Iterative 3D/2D Liver Registration](https://mlsanity.com/paper/2604.10245/) (2026) - [Attenuation-Resilient Alternating Optimization for Laparoscopic Liver Landmark Detection](https://mlsanity.com/paper/2605.26630/) (2026) - [Vis2Reg: Visibility-Aware Landmark-Free Geometric 3D--2D Registration for Liver Laparoscopy](https://mlsanity.com/paper/2607.17810/) (2026) ## Abstract (arXiv) Augmented reality for laparoscopic liver resection is a visualisation mode that allows a surgeon to localise tumours and vessels embedded within the liver by projecting them on top of a laparoscopic image. Preoperative 3D models extracted from CT or MRI data are registered to the intraoperative laparoscopic images during this process. In terms of 3D-2D fusion, most of the algorithms make use of anatomical landmarks to guide registration. These landmarks include the liver's inferior ridge, the falciform ligament, and the occluding contours. They are usually marked by hand in both the laparoscopic image and the 3D model, which is time-consuming and may contain errors if done by a non-experienced user. Therefore, there is a need to automate this process so that augmented reality can be used effectively in the operating room. We present the Preoperative-to-Intraoperative Laparoscopic Fusion Challenge (P2ILF), held during the Medical Imaging and Computer Assisted Interventions (MICCAI 2022) conference, which investigates the possibilities of detecting these landmarks automatically and using them in registration. The challenge was divided into two tasks: 1) A 2D and 3D landmark detection task and 2) a 3D-2D registration task. The teams were provided with training data consisting of 167 laparoscopic images and 9 preoperative 3D models from 9 patients, with the corresponding 2D and 3D landmark annotations. A total of 6 teams from 4 countries participated, whose proposed methods were evaluated on 16 images and two preoperative 3D models from two patients. All the teams proposed deep learning-based methods for the 2D and 3D landmark segmentation tasks and differentiable rendering-based methods for the registration task. Based on the experimental outcomes, we propose three key hypotheses that determine current limitations and future directions for research in this domain.