# Image To Tree with Recursive Prompting

arXiv: https://arxiv.org/abs/2301.00447 · HTML: https://mlsanity.com/paper/2301.00447/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 1 Jan 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation](https://mlsanity.com/paper/1806.02473/) (2018)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Conditional Set Generation with Transformers](https://mlsanity.com/paper/2006.16841/) (2020)
- [TopNet: Topology Preserving Metric Learning for Vessel Tree Reconstruction and Labelling](https://mlsanity.com/paper/2009.08674/) (2020)
- [DeepCenterline: a Multi-task Fully Convolutional Network for Centerline Extraction](https://mlsanity.com/paper/1903.10481/) (2019)
- [Extracting 3D Vascular Structures from Microscopy Images using Convolutional Recurrent Networks](https://mlsanity.com/paper/1705.09597/) (2017)

## Built on by (most cited first, 1 of 1)

- [Cross-domain and Cross-dimension Learning for Image-to-Graph Transformers](https://mlsanity.com/paper/2403.06601/) (2024)

## Abstract (arXiv)

Extracting complex structures from grid-based data is a common key step in automated medical image analysis. The conventional solution to recovering tree-structured geometries typically involves computing the minimal cost path through intermediate representations derived from segmentation masks. However, this methodology has significant limitations in the context of projective imaging of tree-structured 3D anatomical data such as coronary arteries, since there are often overlapping branches in the 2D projection. In this work, we propose a novel approach to predicting tree connectivity structure which reformulates the task as an optimization problem over individual steps of a recursive process. We design and train a two-stage model which leverages the UNet and Transformer architectures and introduces an image-based prompting technique. Our proposed method achieves compelling results on a pair of synthetic datasets, and outperforms a shortest-path baseline.