# ASIST: Annotation-free Synthetic Instance Segmentation and Tracking by Adversarial Simulations

arXiv: https://arxiv.org/abs/2101.00567 · HTML: https://mlsanity.com/paper/2101.00567/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 3 Jan 2021.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks?](https://mlsanity.com/paper/1610.01983/) (2016)
- [Adversarial Synthesis Learning Enables Segmentation Without Target Modality Ground Truth](https://mlsanity.com/paper/1712.07695/) (2017)
- [Extending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS and Gazebo](https://mlsanity.com/paper/1608.05742/) (2016)
- [Instance Segmentation and Tracking with Cosine Embeddings and Recurrent Hourglass Networks](https://mlsanity.com/paper/1806.02070/) (2018)
- [Towards Adversarial Retinal Image Synthesis](https://mlsanity.com/paper/1701.08974/) (2017)
- [Learning Normalized Inputs for Iterative Estimation in Medical Image Segmentation](https://mlsanity.com/paper/1702.05174/) (2017)
- [Geometrically Matched Multi-source Microscopic Image Synthesis Using Bidirectional Adversarial Networks](https://mlsanity.com/paper/2010.13308/) (2020)
- [Unsupervised Learning for Large-Scale Fiber Detection and Tracking in Microscopic Material Images](https://mlsanity.com/paper/1805.10256/) (2018)
- [Decompose to manipulate: Manipulable Object Synthesis in 3D Medical Images with Structured Image Decomposition](https://mlsanity.com/paper/1812.01737/) (2018)
- [Faster Mean-shift: GPU-accelerated clustering for cosine embedding-based cell segmentation and tracking](https://mlsanity.com/paper/2007.14283/) (2020)
- [GAN based Unsupervised Segmentation: Should We Match the Exact Number of Objects](https://mlsanity.com/paper/2010.11438/) (2020)

## Built on by (most cited first, 3 of 3)

- [SynCellFactory: Generative Data Augmentation for Cell Tracking](https://mlsanity.com/paper/2404.16421/) (2024)
- [GPU-accelerated Faster Mean Shift with euclidean distance metrics](https://mlsanity.com/paper/2112.13891/) (2021)
- [Microscopy Image Segmentation via Point and Shape Regularized Data Synthesis](https://mlsanity.com/paper/2308.09835/) (2023)

## Abstract (arXiv)

Background: The quantitative analysis of microscope videos often requires instance segmentation and tracking of cellular and subcellular objects. The traditional method consists of two stages: (1) performing instance object segmentation of each frame, and (2) associating objects frame-by-frame. Recently, pixel-embedding-based deep learning approaches these two steps simultaneously as a single stage holistic solution. In computer vision, annotated training data with consistent segmentation and tracking is resource intensive, the severity of which is multiplied in microscopy imaging due to (1) dense objects (e.g., overlapping or touching), and (2) high dynamics (e.g., irregular motion and mitosis). Adversarial simulations have provided successful solutions to alleviate the lack of such annotations in dynamics scenes in computer vision, such as using simulated environments (e.g., computer games) to train real-world self-driving systems. Methods: In this paper, we propose an annotation-free synthetic instance segmentation and tracking (ASIST) method with adversarial simulation and single-stage pixel-embedding based learning. Contribution: The contribution of this paper is three-fold: (1) the proposed method aggregates adversarial simulations and single-stage pixel-embedding based deep learning; (2) the method is assessed with both the cellular (i.e., HeLa cells) and subcellular (i.e., microvilli) objects; and (3) to the best of our knowledge, this is the first study to explore annotation-free instance segmentation and tracking study for microscope videos. Results: The ASIST method achieved an important step forward, when compared with fully supervised approaches: ASIST shows 7% to 11% higher segmentation, detection and tracking performance on microvilli relative to fully supervised methods, and comparable performance on Hela cell videos.