# Scene Depth Estimation from Traditional Oriental Landscape Paintings

arXiv: https://arxiv.org/abs/2403.03408 · HTML: https://mlsanity.com/paper/2403.03408/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 59 references that are in the map.
- First posted on arXiv 6 Mar 2024.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [BBDM: Image-to-image Translation with Brownian Bridge Diffusion Models](https://mlsanity.com/paper/2205.07680/) (2022)
- [Dual Diffusion Implicit Bridges for Image-to-Image Translation](https://mlsanity.com/paper/2203.08382/) (2022)
- [CLIPstyler: Image Style Transfer with a Single Text Condition](https://mlsanity.com/paper/2112.00374/) (2021)
- [MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimation](https://mlsanity.com/paper/2307.14460/) (2023)
- [Diffusion-based Image Translation using Disentangled Style and Content Representation](https://mlsanity.com/paper/2209.15264/) (2022)
- [Learning Depth with Convolutional Spatial Propagation Network](https://mlsanity.com/paper/1810.02695/) (2018)
- [Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging](https://mlsanity.com/paper/2105.14021/) (2021)
- [Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning](https://mlsanity.com/paper/2205.09542/) (2022)
- [CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not](https://mlsanity.com/paper/2303.13440/) (2023)
- [The Edge of Depth: Explicit Constraints between Segmentation and Depth](https://mlsanity.com/paper/2004.00171/) (2020)
- [Aligning Latent and Image Spaces to Connect the Unconnectable](https://mlsanity.com/paper/2104.06954/) (2021)
- [Learning to Segment Every Thing](https://mlsanity.com/paper/1711.10370/) (2017)
- [CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification](https://mlsanity.com/paper/2204.14244/) (2022)
- [CLIPascene: Scene Sketching with Different Types and Levels of Abstraction](https://mlsanity.com/paper/2211.17256/) (2022)

## Abstract (arXiv)

Scene depth estimation from paintings can streamline the process of 3D sculpture creation so that visually impaired people appreciate the paintings with tactile sense. However, measuring depth of oriental landscape painting images is extremely challenging due to its unique method of depicting depth and poor preservation. To address the problem of scene depth estimation from oriental landscape painting images, we propose a novel framework that consists of two-step Image-to-Image translation method with CLIP-based image matching at the front end to predict the real scene image that best matches with the given oriental landscape painting image. Then, we employ a pre-trained SOTA depth estimation model for the generated real scene image. In the first step, CycleGAN converts an oriental landscape painting image into a pseudo-real scene image. We utilize CLIP to semantically match landscape photo images with an oriental landscape painting image for training CycleGAN in an unsupervised manner. Then, the pseudo-real scene image and oriental landscape painting image are fed into DiffuseIT to predict a final real scene image in the second step. Finally, we measure depth of the generated real scene image using a pre-trained depth estimation model such as MiDaS. Experimental results show that our approach performs well enough to predict real scene images corresponding to oriental landscape painting images. To the best of our knowledge, this is the first study to measure the depth of oriental landscape painting images. Our research potentially assists visually impaired people in experiencing paintings in diverse ways. We will release our code and resulting dataset.