# Hybrid Cost Volume for Memory-Efficient Optical Flow

arXiv: https://arxiv.org/abs/2409.04243 · HTML: https://mlsanity.com/paper/2409.04243/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 6 Sep 2024.

## Builds on (most cited first, 30 of 34)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [Unifying Flow, Stereo and Depth Estimation](https://mlsanity.com/paper/2211.05783/) (2022)
- [Group-wise Correlation Stereo Network](https://mlsanity.com/paper/1903.04025/) (2019)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [Iterative Geometry Encoding Volume for Stereo Matching](https://mlsanity.com/paper/2303.06615/) (2023)
- [SpatialTracker: Tracking Any 2D Pixels in 3D Space](https://mlsanity.com/paper/2404.04319/) (2024)
- [Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo](https://mlsanity.com/paper/2303.01943/) (2023)
- [Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation](https://mlsanity.com/paper/1904.05290/) (2019)
- [Attention Concatenation Volume for Accurate and Efficient Stereo Matching](https://mlsanity.com/paper/2203.02146/) (2022)
- [MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask](https://mlsanity.com/paper/2003.10955/) (2020)
- [Selective-Stereo: Adaptive Frequency Information Selection for Stereo Matching](https://mlsanity.com/paper/2403.00486/) (2024)
- [The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/1905.00737/) (2019)
- [Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation](https://mlsanity.com/paper/1809.05571/) (2018)
- [A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization](https://mlsanity.com/paper/1903.07414/) (2019)
- [PCW-Net: Pyramid Combination and Warping Cost Volume for Stereo Matching](https://mlsanity.com/paper/2006.12797/) (2020)
- [Accurate and Efficient Stereo Matching via Attention Concatenation Volume](https://mlsanity.com/paper/2209.12699/) (2022)
- [High-Resolution Optical Flow from 1D Attention and Correlation](https://mlsanity.com/paper/2104.13918/) (2021)
- [Learning Optical Flow from a Few Matches](https://mlsanity.com/paper/2104.02166/) (2021)
- [Displacement-Invariant Matching Cost Learning for Accurate Optical Flow Estimation](https://mlsanity.com/paper/2010.14851/) (2020)
- [DIP: Deep Inverse Patchmatch for High-Resolution Optical Flow](https://mlsanity.com/paper/2204.00330/) (2022)
- [HDRFlow: Real-Time HDR Video Reconstruction with Large Motions](https://mlsanity.com/paper/2403.03447/) (2024)

## Built on by (most cited first, 6 of 6)

- [WAFT: Warping-Alone Field Transforms for Optical Flow](https://mlsanity.com/paper/2506.21526/) (2025)
- [BANet: Bilateral Aggregation Network for Mobile Stereo Matching](https://mlsanity.com/paper/2503.03259/) (2025)
- [MEMFOF: High-Resolution Training for Memory-Efficient Multi-Frame Optical Flow Estimation](https://mlsanity.com/paper/2506.23151/) (2025)
- [Removing Cost Volumes from Optical Flow Estimators](https://mlsanity.com/paper/2510.13317/) (2025)
- [BAT: Learning Event-based Optical Flow with Bidirectional Adaptive Temporal Correlation](https://mlsanity.com/paper/2503.03256/) (2025)
- [Efficient All-Pairs Correlation Volume Sampling for Optical Flow Estimation](https://mlsanity.com/paper/2505.16942/) (2025)

## Abstract (arXiv)

Current state-of-the-art flow methods are mostly based on dense all-pairs cost volumes. However, as image resolution increases, the computational and spatial complexity of constructing these cost volumes grows at a quartic rate, making these methods impractical for high-resolution images. In this paper, we propose a novel Hybrid Cost Volume for memory-efficient optical flow, named HCV. To construct HCV, we first propose a Top-k strategy to separate the 4D cost volume into two global 3D cost volumes. These volumes significantly reduce memory usage while retaining a substantial amount of matching information. We further introduce a local 4D cost volume with a local search space to supplement the local information for HCV. Based on HCV, we design a memory-efficient optical flow network, named HCVFlow. Compared to the recurrent flow methods based the all-pairs cost volumes, our HCVFlow significantly reduces memory consumption while ensuring high accuracy. We validate the effectiveness and efficiency of our method on the Sintel and KITTI datasets and real-world 4K (2160*3840) resolution images. Extensive experiments show that our HCVFlow has very low memory usage and outperforms other memory-efficient methods in terms of accuracy. The code is publicly available at https://github.com/gangweiX/HCVFlow.