# Training of Convolutional Networks on Multiple Heterogeneous Datasets for Street Scene Semantic Segmentation

arXiv: https://arxiv.org/abs/1803.05675 · HTML: https://mlsanity.com/paper/1803.05675/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2018.

## Built on by (most cited first, 18 of 18)

- [MSeg: A Composite Dataset for Multi-domain Semantic Segmentation](https://mlsanity.com/paper/2112.13762/) (2021)
- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)
- [Multi-Space Alignments Towards Universal LiDAR Segmentation](https://mlsanity.com/paper/2405.01538/) (2024)
- [Dense outlier detection and open-set recognition based on training with noisy negative images](https://mlsanity.com/paper/2101.09193/) (2021)
- [Multi-domain semantic segmentation with overlapping labels](https://mlsanity.com/paper/2108.11224/) (2021)
- [Learning Semantic Segmentation from Multiple Datasets with Label Shifts](https://mlsanity.com/paper/2202.14030/) (2022)
- [Multi-Dataset Co-Training with Sharpness-Aware Optimization for Audio Anti-spoofing](https://mlsanity.com/paper/2305.19953/) (2023)
- [Flattening the Parent Bias: Hierarchical Semantic Segmentation in the Poincar'e Ball](https://mlsanity.com/paper/2404.03778/) (2024)
- [COLA: COarse-LAbel multi-source LiDAR semantic segmentation for autonomous driving](https://mlsanity.com/paper/2311.03017/) (2023)
- [Warp-Refine Propagation: Semi-Supervised Auto-labeling via Cycle-consistency](https://mlsanity.com/paper/2109.13432/) (2021)
- [Automatic universal taxonomies for multi-domain semantic segmentation](https://mlsanity.com/paper/2207.08445/) (2022)
- [Weakly supervised training of universal visual concepts for multi-domain semantic segmentation](https://mlsanity.com/paper/2212.10340/) (2022)
- [COLA: COarse LAbel pre-training for 3D semantic segmentation of sparse LiDAR datasets](https://mlsanity.com/paper/2202.06884/) (2022)
- [Training Semantic Segmentation on Heterogeneous Datasets](https://mlsanity.com/paper/2301.07634/) (2023)
- [Semantic Segmentation on Multiple Visual Domains](https://mlsanity.com/paper/2107.04326/) (2021)
- [Towards Establishing Systematic Classification Requirements for Automated Driving](https://mlsanity.com/paper/2307.14058/) (2023)
- [Online Pseudo-Label Unified Object Detection for Multiple Datasets Training](https://mlsanity.com/paper/2410.15569/) (2024)
- [Knowledge Distillation for Semantic Segmentation: A Label Space Unification Approach](https://mlsanity.com/paper/2502.19177/) (2025)

## Abstract (arXiv)

We propose a convolutional network with hierarchical classifiers for per-pixel semantic segmentation, which is able to be trained on multiple, heterogeneous datasets and exploit their semantic hierarchy. Our network is the first to be simultaneously trained on three different datasets from the intelligent vehicles domain, i.e. Cityscapes, GTSDB and Mapillary Vistas, and is able to handle different semantic level-of-detail, class imbalances, and different annotation types, i.e. dense per-pixel and sparse bounding-box labels. We assess our hierarchical approach, by comparing against flat, non-hierarchical classifiers and we show improvements in mean pixel accuracy of 13.0% for Cityscapes classes and 2.4% for Vistas classes and 32.3% for GTSDB classes. Our implementation achieves inference rates of 17 fps at a resolution of 520x706 for 108 classes running on a GPU.