# ROFT: Real-Time Optical Flow-Aided 6D Object Pose and Velocity Tracking

arXiv: https://arxiv.org/abs/2111.03821 · HTML: https://mlsanity.com/paper/2111.03821/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 6 Nov 2021.

## Builds on (most cited first, 13 of 13)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects](https://mlsanity.com/paper/1701.05498/) (2017)
- [HybridPose: 6D Object Pose Estimation under Hybrid Representations](https://mlsanity.com/paper/2001.01869/) (2020)
- [PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking](https://mlsanity.com/paper/1905.09304/) (2019)
- [se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2007.13866/) (2020)
- [BlenderProc](https://mlsanity.com/paper/1911.01911/) (2019)
- [Robust Ego and Object 6-DoF Motion Estimation and Tracking](https://mlsanity.com/paper/2007.13993/) (2020)

## Built on by (most cited first, 10 of 10)

- [Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Tracking](https://mlsanity.com/paper/2406.04316/) (2024)
- [Collision-aware In-hand 6D Object Pose Estimation using Multiple Vision-based Tactile Sensors](https://mlsanity.com/paper/2301.13667/) (2023)
- [Fusing Visual Appearance and Geometry for Multi-modality 6DoF Object Tracking](https://mlsanity.com/paper/2302.11458/) (2023)
- [6-DoF Object Tracking with Event-based Optical Flow and Frames](https://mlsanity.com/paper/2508.14776/) (2025)
- [Event-based Motion & Appearance Fusion for 6D Object Pose Tracking](https://mlsanity.com/paper/2603.08264/) (2026)
- [INV-Flow2PoseNet: Light-Resistant Rigid Object Pose from Optical Flow of RGB-D Images using Images, Normals and Vertices](https://mlsanity.com/paper/2209.06562/) (2022)
- [Nonlinear Motion-Guided and Spatio-Temporal Aware Network for Unsupervised Event-Based Optical Flow](https://mlsanity.com/paper/2505.05089/) (2025)
- [Color-Pair Guided Robust Zero-Shot 6D Pose Estimation and Tracking of Cluttered Objects on Edge Devices](https://mlsanity.com/paper/2509.23647/) (2025)
- [Robust 6-DoF Object Pose Tracking with Built-In Recovery under Occlusions and Rapid Object Motions](https://mlsanity.com/paper/2607.23468/) (2026)
- [ASTRA: Asynchronous Spatio-Temporal Reconstruction via Trajectory Alignment](https://mlsanity.com/paper/2608.02006/) (2026)

## Abstract (arXiv)

6D object pose tracking has been extensively studied in the robotics and computer vision communities. The most promising solutions, leveraging on deep neural networks and/or filtering and optimization, exhibit notable performance on standard benchmarks. However, to our best knowledge, these have not been tested thoroughly against fast object motions. Tracking performance in this scenario degrades significantly, especially for methods that do not achieve real-time performance and introduce non negligible delays. In this work, we introduce ROFT, a Kalman filtering approach for 6D object pose and velocity tracking from a stream of RGB-D images. By leveraging real-time optical flow, ROFT synchronizes delayed outputs of low frame rate Convolutional Neural Networks for instance segmentation and 6D object pose estimation with the RGB-D input stream to achieve fast and precise 6D object pose and velocity tracking. We test our method on a newly introduced photorealistic dataset, Fast-YCB, which comprises fast moving objects from the YCB model set, and on the dataset for object and hand pose estimation HO-3D. Results demonstrate that our approach outperforms state-of-the-art methods for 6D object pose tracking, while also providing 6D object velocity tracking. A video showing the experiments is provided as supplementary material.