# Deep Laparoscopic Stereo Matching with Transformers

arXiv: https://arxiv.org/abs/2207.12152 · HTML: https://mlsanity.com/paper/2207.12152/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 25 Jul 2022.

## Builds on (most cited first, 24 of 24)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [cosFormer: Rethinking Softmax in Attention](https://mlsanity.com/paper/2202.08791/) (2022)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [Hierarchical Neural Architecture Search for Deep Stereo Matching](https://mlsanity.com/paper/2010.13501/) (2020)
- [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021)
- [Displacement-Invariant Matching Cost Learning for Accurate Optical Flow Estimation](https://mlsanity.com/paper/2010.14851/) (2020)
- [E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception](https://mlsanity.com/paper/2107.00229/) (2021)
- [Deep Two-View Structure-from-Motion Revisited](https://mlsanity.com/paper/2104.00556/) (2021)
- [Self-Supervised Siamese Learning on Stereo Image Pairs for Depth Estimation in Robotic Surgery](https://mlsanity.com/paper/1705.08260/) (2017)
- [Self-Supervised Learning for Stereo Matching with Self-Improving Ability](https://mlsanity.com/paper/1709.00930/) (2017)
- [Noise-Aware Unsupervised Deep Lidar-Stereo Fusion](https://mlsanity.com/paper/1904.03868/) (2019)
- [Unsupervised Deep Epipolar Flow for Stationary or Dynamic Scenes](https://mlsanity.com/paper/1904.03848/) (2019)
- [Vicinity Vision Transformer](https://mlsanity.com/paper/2206.10552/) (2022)
- [Self-Supervised Generative Adversarial Network for Depth Estimation in Laparoscopic Images](https://mlsanity.com/paper/2107.04644/) (2021)
- [Open-World Stereo Video Matching with Deep RNN](https://mlsanity.com/paper/1808.03959/) (2018)
- [Stereo Computation for a Single Mixture Image](https://mlsanity.com/paper/1808.08690/) (2018)
- [3D Geometry-Aware Semantic Labeling of Outdoor Street Scenes](https://mlsanity.com/paper/1808.04028/) (2018)

## Built on by (most cited first, 9 of 9)

- [The Devil in Linear Transformer](https://mlsanity.com/paper/2210.10340/) (2022)
- [EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos](https://mlsanity.com/paper/2307.11307/) (2023)
- [SurgicalGS: Dynamic 3D Gaussian Splatting for Accurate Robotic-Assisted Surgical Scene Reconstruction](https://mlsanity.com/paper/2410.09292/) (2024)
- [Linear Video Transformer with Feature Fixation](https://mlsanity.com/paper/2210.08164/) (2022)
- [Robust Real-Time Endoscopic Stereo Matching under Fuzzy Tissue Boundaries](https://mlsanity.com/paper/2503.00731/) (2025)
- [StereoMamba: Real-time and Robust Intraoperative Stereo Disparity Estimation via Long-range Spatial Dependencies](https://mlsanity.com/paper/2504.17401/) (2025)
- [Monocular Depth Guided Occlusion-Aware Disparity Refinement via Semi-supervised Learning in Laparoscopic Images](https://mlsanity.com/paper/2505.08178/) (2025)
- [SurgiATM: A Physics-Guided Plug-and-Play Model for Deep Learning-Based Smoke Removal in Laparoscopic Surgery](https://mlsanity.com/paper/2511.05059/) (2025)
- [MV-dVRK: A Multi-Viewpoint Benchmark for Spatial Surgical Perception](https://mlsanity.com/paper/2609.02717/) (2026)

## Abstract (arXiv)

The self-attention mechanism, successfully employed with the transformer structure is shown promise in many computer vision tasks including image recognition, and object detection. Despite the surge, the use of the transformer for the problem of stereo matching remains relatively unexplored. In this paper, we comprehensively investigate the use of the transformer for the problem of stereo matching, especially for laparoscopic videos, and propose a new hybrid deep stereo matching framework (HybridStereoNet) that combines the best of the CNN and the transformer in a unified design. To be specific, we investigate several ways to introduce transformers to volumetric stereo matching pipelines by analyzing the loss landscape of the designs and in-domain/cross-domain accuracy. Our analysis suggests that employing transformers for feature representation learning, while using CNNs for cost aggregation will lead to faster convergence, higher accuracy and better generalization than other options. Our extensive experiments on Sceneflow, SCARED2019 and dVPN datasets demonstrate the superior performance of our HybridStereoNet.