# Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors

arXiv: https://arxiv.org/abs/2411.17249 · HTML: https://mlsanity.com/paper/2411.17249/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 58 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 30 of 41)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](https://mlsanity.com/paper/2110.04994/) (2021)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image](https://mlsanity.com/paper/2403.12013/) (2024)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos](https://mlsanity.com/paper/2409.02095/) (2024)
- [ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals](https://mlsanity.com/paper/1905.02082/) (2019)
- [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://mlsanity.com/paper/2311.16933/) (2023)
- [CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation](https://mlsanity.com/paper/2406.02509/) (2024)
- [Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction](https://mlsanity.com/paper/2409.18124/) (2024)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [ARF: Artistic Radiance Fields](https://mlsanity.com/paper/2206.06360/) (2022)
- [State of the Art on Diffusion Models for Visual Computing](https://mlsanity.com/paper/2310.07204/) (2023)
- [Rethinking Inductive Biases for Surface Normal Estimation](https://mlsanity.com/paper/2403.00712/) (2024)
- [3D Common Corruptions and Data Augmentation](https://mlsanity.com/paper/2203.01441/) (2022)

## Built on by (most cited first, 11 of 11)

- [NormalCrafter: Learning Temporally Consistent Normals from Video Diffusion Priors](https://mlsanity.com/paper/2504.11427/) (2025)
- [FlashDepth: Real-time Streaming Video Depth Estimation at 2K Resolution](https://mlsanity.com/paper/2504.07093/) (2025)
- [Video Generation Models are General-Purpose Vision Learners](https://mlsanity.com/paper/2607.09024/) (2026)
- [THFM: A Unified Video Foundation Model for 4D Human Perception and Beyond](https://mlsanity.com/paper/2603.25892/) (2026)
- [BeltCrack: the First Sequential-image Industrial Conveyor Belt Crack Detection Dataset and Its Baseline with Triple-domain Feature Learning](https://mlsanity.com/paper/2506.17892/) (2025)
- [Towards Depth Foundation Model: Recent Trends in Vision-Based Depth Estimation](https://mlsanity.com/paper/2507.11540/) (2025)
- [Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos](https://mlsanity.com/paper/2603.12064/) (2026)
- [AsyncMDE: Real-Time Monocular Depth Estimation via Asynchronous Spatial Memory](https://mlsanity.com/paper/2603.10438/) (2026)
- [GemDepth: Geometry-Embedded Features for 3D-Consistent Video Depth](https://mlsanity.com/paper/2605.10525/) (2026)
- [Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis](https://mlsanity.com/paper/2606.01590/) (2026)
- [ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning](https://mlsanity.com/paper/2607.01677/) (2026)

## Abstract (arXiv)

We present Buffer Anytime, a framework for estimation of depth and normal maps (which we call geometric buffers) from video that eliminates the need for paired video--depth and video--normal training data. Instead of relying on large-scale annotated video datasets, we demonstrate high-quality video buffer estimation by leveraging single-image priors with temporal consistency constraints. Our zero-shot training strategy combines state-of-the-art image estimation models based on optical flow smoothness through a hybrid loss function, implemented via a lightweight temporal attention architecture. Applied to leading image models like Depth Anything V2 and Marigold-E2E-FT, our approach significantly improves temporal consistency while maintaining accuracy. Experiments show that our method not only outperforms image-based approaches but also achieves results comparable to state-of-the-art video models trained on large-scale paired video datasets, despite using no such paired video data.