# Cross-domain and Cross-dimension Learning for Image-to-Graph Transformers

arXiv: https://arxiv.org/abs/2403.06601 · HTML: https://mlsanity.com/paper/2403.06601/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 58 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [Domain Adaptive Faster R-CNN for Object Detection in the Wild](https://mlsanity.com/paper/1803.03243/) (2018)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [Classes Matter: A Fine-grained Adversarial Approach to Cross-domain Semantic Segmentation](https://mlsanity.com/paper/2007.09222/) (2020)
- [Benchmarking Detection Transfer Learning with Vision Transformers](https://mlsanity.com/paper/2111.11429/) (2021)
- [RoadTracer: Automatic Extraction of Road Networks from Aerial Images](https://mlsanity.com/paper/1802.03680/) (2018)
- [Graph Representation Learning in Biomedicine](https://mlsanity.com/paper/2104.04883/) (2021)
- [ROSE: A Retinal OCT-Angiography Vessel Segmentation Dataset and New Model](https://mlsanity.com/paper/2007.05201/) (2020)
- [Relationformer: A Unified Framework for Image-to-Graph Generation](https://mlsanity.com/paper/2203.10202/) (2022)
- [DeepVesselNet: Vessel Segmentation, Centerline Prediction, and Bifurcation Detection in 3-D Angiographic Volumes](https://mlsanity.com/paper/1803.09340/) (2018)
- [CGC-Net: Cell Graph Convolutional Network for Grading of Colorectal Cancer Histology Images](https://mlsanity.com/paper/1909.01068/) (2019)
- [Sat2Graph: Road Graph Extraction through Graph-Tensor Encoding](https://mlsanity.com/paper/2007.09547/) (2020)
- [RNGDet++: Road Network Graph Detection by Transformer with Instance Segmentation and Multi-scale Features Enhancement](https://mlsanity.com/paper/2209.10150/) (2022)
- [Image-Conditioned Graph Generation for Road Network Extraction](https://mlsanity.com/paper/1910.14388/) (2019)
- [Scalable Robust Graph and Feature Extraction for Arbitrary Vessel Networks in Large Volumetric Datasets](https://mlsanity.com/paper/2102.03444/) (2021)
- [Physiology-based simulation of the retinal vasculature enables annotation-free segmentation of OCT angiographs](https://mlsanity.com/paper/2207.11102/) (2022)
- [Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer?](https://mlsanity.com/paper/2209.07026/) (2022)
- [Revisiting DETR Pre-training for Object Detection](https://mlsanity.com/paper/2308.01300/) (2023)
- [simCrossTrans: A Simple Cross-Modality Transfer Learning for Object Detection with ConvNets or Vision Transformers](https://mlsanity.com/paper/2203.10456/) (2022)
- [Image To Tree with Recursive Prompting](https://mlsanity.com/paper/2301.00447/) (2023)

## Built on by (most cited first, 2 of 2)

- [VisAdj: Learning Adjacency Matrices from Node-Link Images](https://mlsanity.com/paper/2608.21825/) (2026)
- [Temporally Consistent Graph Extraction and Matching for Longitudinal Angiographic Images](https://mlsanity.com/paper/2609.16889/) (2026)

## Abstract (arXiv)

Direct image-to-graph transformation is a challenging task that involves solving object detection and relationship prediction in a single model. Due to this task's complexity, large training datasets are rare in many domains, making the training of deep-learning methods challenging. This data sparsity necessitates transfer learning strategies akin to the state-of-the-art in general computer vision. In this work, we introduce a set of methods enabling cross-domain and cross-dimension learning for image-to-graph transformers. We propose (1) a regularized edge sampling loss to effectively learn object relations in multiple domains with different numbers of edges, (2) a domain adaptation framework for image-to-graph transformers aligning image- and graph-level features from different domains, and (3) a projection function that allows using 2D data for training 3D transformers. We demonstrate our method's utility in cross-domain and cross-dimension experiments, where we utilize labeled data from 2D road networks for simultaneous learning in vastly different target domains. Our method consistently outperforms standard transfer learning and self-supervised pretraining on challenging benchmarks, such as retinal or whole-brain vessel graph extraction.