# PCDepth: Pattern-based Complementary Learning for Monocular Depth Estimation by Best of Both Worlds

arXiv: https://arxiv.org/abs/2402.18925 · HTML: https://mlsanity.com/paper/2402.18925/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 43 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion](https://mlsanity.com/paper/1812.08156/) (2018)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [The Multi Vehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception](https://mlsanity.com/paper/1801.10202/) (2018)
- [End-to-End Learning of Representations for Asynchronous Event-Based Data](https://mlsanity.com/paper/1904.08245/) (2019)
- [Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlation](https://mlsanity.com/paper/2203.11483/) (2022)
- [BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation](https://mlsanity.com/paper/2204.00987/) (2022)
- [NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation](https://mlsanity.com/paper/2203.01502/) (2022)
- [LocalBins: Improving Depth Estimation by Learning Local Distributions](https://mlsanity.com/paper/2203.15132/) (2022)
- [Event-aided Direct Sparse Odometry](https://mlsanity.com/paper/2204.07640/) (2022)
- [Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction](https://mlsanity.com/paper/2102.09320/) (2021)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [Learning Monocular Dense Depth from Events](https://mlsanity.com/paper/2010.08350/) (2020)
- [Time-Ordered Recent Event (TORE) Volumes for Event Cameras](https://mlsanity.com/paper/2103.06108/) (2021)
- [iDisc: Internal Discretization for Monocular Depth Estimation](https://mlsanity.com/paper/2304.06334/) (2023)
- [Hierarchical Neural Memory Network for Low Latency Event Processing](https://mlsanity.com/paper/2305.17852/) (2023)
- [EVEN: An Event-Based Framework for Monocular Depth Estimation at Adverse Night Conditions](https://mlsanity.com/paper/2302.03860/) (2023)
- [Event-based Monocular Dense Depth Estimation with Recurrent Transformers](https://mlsanity.com/paper/2212.02791/) (2022)

## Built on by (most cited first, 7 of 7)

- [TESPEC: Temporally-Enhanced Self-Supervised Pretraining for Event Cameras](https://mlsanity.com/paper/2508.00913/) (2025)
- [FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion](https://mlsanity.com/paper/2503.19739/) (2025)
- [DELTA: Dense Depth from Events and LiDAR using Transformer's Attention](https://mlsanity.com/paper/2505.02593/) (2025)
- [Temporal-Guided Visual Foundation Models for Event-Based Vision](https://mlsanity.com/paper/2511.06238/) (2025)
- [LiFR v2: Completion-Augmented Event Propagation for High-Rate Dense Prediction](https://mlsanity.com/paper/2609.25803/) (2026)
- [Depth Hypothesis Guided Iterative Refinement for Event-Image Monocular Depth Estimation](https://mlsanity.com/paper/2610.03439/) (2026)
- [UniDynamics: Event-RGB Fusion for Unified Future 4D Dynamic Scene Generation](https://mlsanity.com/paper/2610.03473/) (2026)

## Abstract (arXiv)

Event cameras can record scene dynamics with high temporal resolution, providing rich scene details for monocular depth estimation (MDE) even at low-level illumination. Therefore, existing complementary learning approaches for MDE fuse intensity information from images and scene details from event data for better scene understanding. However, most methods directly fuse two modalities at pixel level, ignoring that the attractive complementarity mainly impacts high-level patterns that only occupy a few pixels. For example, event data is likely to complement contours of scene objects. In this paper, we discretize the scene into a set of high-level patterns to explore the complementarity and propose a Pattern-based Complementary learning architecture for monocular Depth estimation (PCDepth). Concretely, PCDepth comprises two primary components: a complementary visual representation learning module for discretizing the scene into high-level patterns and integrating complementary patterns across modalities and a refined depth estimator aimed at scene reconstruction and depth prediction while maintaining an efficiency-accuracy balance. Through pattern-based complementary learning, PCDepth fully exploits two modalities and achieves more accurate predictions than existing methods, especially in challenging nighttime scenarios. Extensive experiments on MVSEC and DSEC datasets verify the effectiveness and superiority of our PCDepth. Remarkably, compared with state-of-the-art, PCDepth achieves a 37.9% improvement in accuracy in MVSEC nighttime scenarios.