# Cross-Dataset Collaborative Learning for Semantic Segmentation in Autonomous Driving

arXiv: https://arxiv.org/abs/2103.11351 · HTML: https://mlsanity.com/paper/2103.11351/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 21 Mar 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1606.02147/) (2016)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Domain-Specific Batch Normalization for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1906.03950/) (2019)
- [ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation](https://mlsanity.com/paper/1803.06815/) (2018)
- [The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation](https://mlsanity.com/paper/1611.09326/) (2016)
- [Boundary-Aware Feature Propagation for Scene Segmentation](https://mlsanity.com/paper/1909.00179/) (2019)
- [ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network](https://mlsanity.com/paper/1811.11431/) (2018)
- [Towards Universal Object Detection by Domain Attention](https://mlsanity.com/paper/1904.04402/) (2019)
- [Multi-source Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/1910.12181/) (2019)

## Built on by (most cited first, 23 of 23)

- [Large-scale Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2106.03149/) (2021)
- [Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training](https://mlsanity.com/paper/2308.09718/) (2023)
- [AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weeks](https://mlsanity.com/paper/2305.09666/) (2023)
- [Multi-Domain Incremental Learning for Semantic Segmentation](https://mlsanity.com/paper/2110.12205/) (2021)
- [Label-Assemble: Leveraging Multiple Datasets with Partial Labels](https://mlsanity.com/paper/2109.12265/) (2021)
- [LMSeg: Language-guided Multi-dataset Segmentation](https://mlsanity.com/paper/2302.13495/) (2023)
- [I3CL:Intra- and Inter-Instance Collaborative Learning for Arbitrary-shaped Scene Text Detection](https://mlsanity.com/paper/2108.01343/) (2021)
- [Learning Semantic Segmentation from Multiple Datasets with Label Shifts](https://mlsanity.com/paper/2202.14030/) (2022)
- [Semi-Supervised Semantic Segmentation Based on Pseudo-Labels: A Survey](https://mlsanity.com/paper/2403.01909/) (2024)
- [Reflectivity Is All You Need!: Advancing LiDAR Semantic Segmentation](https://mlsanity.com/paper/2403.13188/) (2024)
- [U-MixFormer: UNet-like Transformer with Mix-Attention for Efficient Semantic Segmentation](https://mlsanity.com/paper/2312.06272/) (2023)
- [GeoAuxNet: Towards Universal 3D Representation Learning for Multi-sensor Point Clouds](https://mlsanity.com/paper/2403.19220/) (2024)
- [Training Semantic Segmentation on Heterogeneous Datasets](https://mlsanity.com/paper/2301.07634/) (2023)
- [Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNs](https://mlsanity.com/paper/2407.10534/) (2024)
- [Active Semantic Localization with Graph Neural Embedding](https://mlsanity.com/paper/2305.06141/) (2023)
- [Representation Learning for the Automatic Indexing of Sound Effects Libraries](https://mlsanity.com/paper/2208.09096/) (2022)
- [MetaMorphosis: Task-oriented Privacy Cognizant Feature Generation for Multi-task Learning](https://mlsanity.com/paper/2305.07815/) (2023)
- [A Decoding Scheme with Successive Aggregation of Multi-Level Features for Light-Weight Semantic Segmentation](https://mlsanity.com/paper/2402.11201/) (2024)
- [Knowledge Distillation for Semantic Segmentation: A Label Space Unification Approach](https://mlsanity.com/paper/2502.19177/) (2025)
- [Multi-Disease-Aware Training Strategy for Cardiac MR Image Segmentation](https://mlsanity.com/paper/2503.17896/) (2025)
- [Self-Supervised Pre-training with Combined Datasets for 3D Perception in Autonomous Driving](https://mlsanity.com/paper/2504.12709/) (2025)
- [ShapeKit](https://mlsanity.com/paper/2506.24003/) (2025)
- [SynSeg: Feature Synergy for Multi-Category Contrastive Learning in End-to-End Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2508.06115/) (2025)

## Abstract (arXiv)

Semantic segmentation is an important task for scene understanding in self-driving cars and robotics, which aims to assign dense labels for all pixels in the image. Existing work typically improves semantic segmentation performance by exploring different network architectures on a target dataset. Little attention has been paid to build a unified system by simultaneously learning from multiple datasets due to the inherent distribution shift across different datasets. In this paper, we propose a simple, flexible, and general method for semantic segmentation, termed Cross-Dataset Collaborative Learning (CDCL). Our goal is to train a unified model for improving the performance in each dataset by leveraging information from all the datasets. Specifically, we first introduce a family of Dataset-Aware Blocks (DAB) as the fundamental computing units of the network, which help capture homogeneous convolutional representations and heterogeneous statistics across different datasets. Second, we present a Dataset Alternation Training (DAT) mechanism to facilitate the collaborative optimization procedure. We conduct extensive evaluations on diverse semantic segmentation datasets for autonomous driving. Experiments demonstrate that our method consistently achieves notable improvements over prior single-dataset and cross-dataset training methods without introducing extra FLOPs. Particularly, with the same architecture of PSPNet (ResNet-18), our method outperforms the single-dataset baseline by 5.65%, 6.57%, and 5.79% mIoU on the validation sets of Cityscapes, BDD100K, CamVid, respectively. We also apply CDCL for point cloud 3D semantic segmentation and achieve improved performance, which further validates the superiority and generality of our method. Code and models will be released.