# A Hybrid mmWave and Camera System for Long-Range Depth Imaging

arXiv: https://arxiv.org/abs/2106.07856 · HTML: https://mlsanity.com/paper/2106.07856/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 71 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 6 of 6)

- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection](https://mlsanity.com/paper/2005.07431/) (2020)
- [Monocular Depth Estimation Based On Deep Learning: An Overview](https://mlsanity.com/paper/2003.06620/) (2020)
- [A Benchmark for Lidar Sensors in Fog: Is Detection Breaking Down?](https://mlsanity.com/paper/1912.03251/) (2019)
- [On the Importance of Stereo for Accurate Depth Estimation: An Efficient Semi-Supervised Deep Neural Network Approach](https://mlsanity.com/paper/1803.09719/) (2018)
- [Kinect Range Sensing: Structured-Light versus Time-of-Flight Kinect](https://mlsanity.com/paper/1505.05459/) (2015)

## Abstract (arXiv)

mmWave radars offer excellent depth resolution even at very long ranges owing to their high bandwidth. But their angular resolution is at least an order-of-magnitude worse than camera and lidar systems. Hence, mmWave radar is not a capable 3-D imaging solution in isolation. We propose Metamoran, a system that combines the complimentary strengths of radar and camera to obtain accurate, high resolution depth images over long ranges even in high clutter environments, all from a single fixed vantage point. Metamoran enables rich long-range depth imaging with applications in security and surveillance, roadside safety infrastructure and wide-area mapping. Our approach leverages the high angular resolution from cameras using computer vision techniques, including image segmentation and monocular depth estimation, to obtain object shape. Our core contribution is a method to convert this object shape into an RF I/Q equivalent, which we use in a novel radar processing pipeline to help declutter the scene and capture extremely weak reflections from objects at long distances. We perform a detailed evaluation of Metamoran's depth imaging capabilities in 400 diverse scenes. Our evaluation shows that Metamoran estimates the depth of static objects up to 90 m and moving objects up to 305 m and with a median error of 28 cm, an improvement of 13× compared to a naive radar+camera baseline and 23× compared to monocular depth estimation.