# Towards High-Frequency Tracking and Fast Edge-Aware Optimization

arXiv: https://arxiv.org/abs/2309.00777 · HTML: https://mlsanity.com/paper/2309.00777/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 159 references that are in the map.
- First posted on arXiv 2 Sep 2023.

## Builds on (most cited first, 17 of 17)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Deep Bilateral Learning for Real-Time Image Enhancement](https://mlsanity.com/paper/1707.02880/) (2017)
- [The Fast Bilateral Solver](https://mlsanity.com/paper/1511.03296/) (2015)
- [EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow](https://mlsanity.com/paper/1501.02565/) (2015)
- [Learning Affinity via Spatial Propagation Networks](https://mlsanity.com/paper/1710.01020/) (2017)
- [Fast End-to-End Trainable Guided Filter](https://mlsanity.com/paper/1803.05619/) (2018)
- [Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform](https://mlsanity.com/paper/1511.03328/) (2015)
- [Rolling-Shutter-Aware Differential SfM and Image Rectification](https://mlsanity.com/paper/1903.03943/) (2019)
- [Rolling Shutter Camera Relative Pose: Generalized Epipolar Geometry](https://mlsanity.com/paper/1605.00475/) (2016)
- [Minimal Solvers for Monocular Rolling Shutter Compensation under Ackermann Motion](https://mlsanity.com/paper/1712.03159/) (2017)
- [Self-calibration-based Approach to Critical Motion Sequences of Rolling-shutter Structure from Motion](https://mlsanity.com/paper/1611.05476/) (2016)
- [Gyroscope-aided Relative Pose Estimation for Rolling Shutter Cameras](https://mlsanity.com/paper/1904.06770/) (2019)
- [Inertial-aided Rolling Shutter Relative Pose Estimation](https://mlsanity.com/paper/1712.00184/) (2017)
- [Rolling Shutter Camera Synchronization with Sub-millisecond Accuracy](https://mlsanity.com/paper/1902.11084/) (2019)
- [The Domain Transform Solver](https://mlsanity.com/paper/1805.04590/) (2018)

## Abstract (arXiv)

This dissertation advances the state of the art for AR/VR tracking systems by increasing the tracking frequency by orders of magnitude and proposes an efficient algorithm for the problem of edge-aware optimization. AR/VR is a natural way of interacting with computers, where the physical and digital worlds coexist. We are on the cusp of a radical change in how humans perform and interact with computing. Humans are sensitive to small misalignments between the real and the virtual world, and tracking at kilo-Hertz frequencies becomes essential. Current vision-based systems fall short, as their tracking frequency is implicitly limited by the frame-rate of the camera. This thesis presents a prototype system which can track at orders of magnitude higher than the state-of-the-art methods using multiple commodity cameras. The proposed system exploits characteristics of the camera traditionally considered as flaws, namely rolling shutter and radial distortion. The experimental evaluation shows the effectiveness of the method for various degrees of motion. Furthermore, edge-aware optimization is an indispensable tool in the computer vision arsenal for accurate filtering of depth-data and image-based rendering, which is increasingly being used for content creation and geometry processing for AR/VR. As applications increasingly demand higher resolution and speed, there exists a need to develop methods that scale accordingly. This dissertation proposes such an edge-aware optimization framework which is efficient, accurate, and algorithmically scales well, all of which are much desirable traits not found jointly in the state of the art. The experiments show the effectiveness of the framework in a multitude of computer vision tasks such as computational photography and stereo.