# Towards Real-Time Generation of Delay-Compensated Video Feeds for Outdoor Mobile Robot Teleoperation

arXiv: https://arxiv.org/abs/2409.09921 · HTML: https://mlsanity.com/paper/2409.09921/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 72 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Builds on (most cited first, 30 of 37)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Neural 3D Mesh Renderer](https://mlsanity.com/paper/1711.07566/) (2017)
- [Gaussian Splatting SLAM](https://mlsanity.com/paper/2312.06741/) (2023)
- [Compact 3D Gaussian Representation for Radiance Field](https://mlsanity.com/paper/2311.13681/) (2023)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [Video Frame Synthesis using Deep Voxel Flow](https://mlsanity.com/paper/1702.02463/) (2017)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS](https://mlsanity.com/paper/2312.04564/) (2023)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [Single-Image Depth Perception in the Wild](https://mlsanity.com/paper/1604.03901/) (2016)
- [Differentiable Rendering: A Survey](https://mlsanity.com/paper/2006.12057/) (2020)
- [Stochastic Latent Residual Video Prediction](https://mlsanity.com/paper/2002.09219/) (2020)
- [Dual Motion GAN for Future-Flow Embedded Video Prediction](https://mlsanity.com/paper/1708.00284/) (2017)
- [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://mlsanity.com/paper/2303.09875/) (2023)
- [Predicting Deeper into the Future of Semantic Segmentation](https://mlsanity.com/paper/1703.07684/) (2017)
- [Teleoperation of Humanoid Robots: A Survey](https://mlsanity.com/paper/2301.04317/) (2023)
- [WayFAST: Navigation with Predictive Traversability in the Field](https://mlsanity.com/paper/2203.12071/) (2022)
- [Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis](https://mlsanity.com/paper/2406.06216/) (2024)
- [Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer](https://mlsanity.com/paper/1903.11149/) (2019)
- [Learned Visual Navigation for Under-Canopy Agricultural Robots](https://mlsanity.com/paper/2107.02792/) (2021)

## Abstract (arXiv)

Teleoperation is an important technology to enable supervisors to control agricultural robots remotely. However, environmental factors in dense crop rows and limitations in network infrastructure hinder the reliability of data streamed to teleoperators. These issues result in delayed and variable frame rate video feeds that often deviate significantly from the robot's actual viewpoint. We propose a modular learning-based vision pipeline to generate delay-compensated images in real-time for supervisors. Our extensive offline evaluations demonstrate that our method generates more accurate images compared to state-of-the-art approaches in our setting. Additionally, ours is one of the few works to evaluate a delay-compensation method in outdoor field environments with complex terrain on data from a real robot in real-time. Resulting videos and code are provided at https://sites.google.com/illinois.edu/comp-teleop.