# A Pipeline for Vision-Based On-Orbit Proximity Operations Using Deep Learning and Synthetic Imagery

arXiv: https://arxiv.org/abs/2101.05661 · HTML: https://mlsanity.com/paper/2101.05661/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 36 references that are in the map.
- First posted on arXiv 14 Jan 2021.

## Builds on (most cited first, 8 of 8)

- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Speed/accuracy trade-offs for modern convolutional object detectors](https://mlsanity.com/paper/1611.10012/) (2016)
- [3D Face Reconstruction by Learning from Synthetic Data](https://mlsanity.com/paper/1609.04387/) (2016)
- [Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering](https://mlsanity.com/paper/1907.04298/) (2019)
- [On Pre-Trained Image Features and Synthetic Images for Deep Learning](https://mlsanity.com/paper/1710.10710/) (2017)
- [Pose Estimation for Non-Cooperative Spacecraft Rendezvous Using Convolutional Neural Networks](https://mlsanity.com/paper/1809.07238/) (2018)
- [Towards Robust Learning-Based Pose Estimation of Noncooperative Spacecraft](https://mlsanity.com/paper/1909.00392/) (2019)

## Built on by (most cited first, 2 of 2)

- [Deep Learning for Real Time Satellite Pose Estimation on Low Power Edge TPU](https://mlsanity.com/paper/2204.03296/) (2022)
- [Machine Learning Systems: A Survey from a Data-Oriented Perspective](https://mlsanity.com/paper/2302.04810/) (2023)

## Abstract (arXiv)

Deep learning has become the gold standard for image processing over the past decade. Simultaneously, we have seen growing interest in orbital activities such as satellite servicing and debris removal that depend on proximity operations between spacecraft. However, two key challenges currently pose a major barrier to the use of deep learning for vision-based on-orbit proximity operations. Firstly, efficient implementation of these techniques relies on an effective system for model development that streamlines data curation, training, and evaluation. Secondly, a scarcity of labeled training data (images of a target spacecraft) hinders creation of robust deep learning models. This paper presents an open-source deep learning pipeline, developed specifically for on-orbit visual navigation applications, that addresses these challenges. The core of our work consists of two custom software tools built on top of a cloud architecture that interconnects all stages of the model development process. The first tool leverages Blender, an open-source 3D graphics toolset, to generate labeled synthetic training data with configurable model poses (positions and orientations), lighting conditions, backgrounds, and commonly observed in-space image aberrations. The second tool is a plugin-based framework for effective dataset curation and model training; it provides common functionality like metadata generation and remote storage access to all projects while giving complete independence to project-specific code. Time-consuming, graphics-intensive processes such as synthetic image generation and model training run on cloud-based computational resources which scale to any scope and budget and allow development of even the largest datasets and models from any machine. The presented system has been used in the Texas Spacecraft Laboratory with marked benefits in development speed and quality.