# Learning CNN on ViT: A Hybrid Model to Explicitly Class-specific Boundaries for Domain Adaptation

arXiv: https://arxiv.org/abs/2403.18360 · HTML: https://mlsanity.com/paper/2403.18360/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Semi-supervised Domain Adaptation via Minimax Entropy](https://mlsanity.com/paper/1904.06487/) (2019)
- [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019)
- [On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location](https://mlsanity.com/paper/2003.07064/) (2020)
- [Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations](https://mlsanity.com/paper/2003.12237/) (2020)
- [Gradually Vanishing Bridge for Adversarial Domain Adaptation](https://mlsanity.com/paper/2003.13183/) (2020)
- [FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2011.09230/) (2020)
- [SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2012.11460/) (2020)
- [Domain Adaptation with Auxiliary Target Domain-Oriented Classifier](https://mlsanity.com/paper/2007.04171/) (2020)
- [Attract, Perturb, and Explore: Learning a Feature Alignment Network for Semi-supervised Domain Adaptation](https://mlsanity.com/paper/2007.09375/) (2020)
- [Cross-Domain Adaptive Clustering for Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2104.09415/) (2021)
- [Reusing the Task-specific Classifier as a Discriminator: Discriminator-free Adversarial Domain Adaptation](https://mlsanity.com/paper/2204.03838/) (2022)
- [Domain Conditioned Adaptation Network](https://mlsanity.com/paper/2005.06717/) (2020)
- [Semantic Concentration for Domain Adaptation](https://mlsanity.com/paper/2108.05720/) (2021)
- [Deep Co-Training with Task Decomposition for Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2007.12684/) (2020)
- [Rethinking Distributional Matching Based Domain Adaptation](https://mlsanity.com/paper/2006.13352/) (2020)
- [Semi-Supervised Domain Adaptation with Source Label Adaptation](https://mlsanity.com/paper/2302.02335/) (2023)
- [Semi-supervised Domain Adaptation via Sample-to-Sample Self-Distillation](https://mlsanity.com/paper/2111.14353/) (2021)
- [Gradual Source Domain Expansion for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2311.09599/) (2023)
- [Semi-supervised Domain Adaptive Structure Learning](https://mlsanity.com/paper/2112.06161/) (2021)
- [Semi-supervised Domain Adaptation via Prototype-based Multi-level Learning](https://mlsanity.com/paper/2305.02693/) (2023)

## Built on by (most cited first, 12 of 12)

- [Efficient unsupervised domain adaptation via self-supervised vision transformer and synergistic cross-domain alignment](https://mlsanity.com/paper/2407.21311/) (2024)
- [CTA-Net: A CNN-Transformer Aggregation Network for Improving Multi-Scale Feature Extraction](https://mlsanity.com/paper/2410.11428/) (2024)
- [UPRE: Zero-Shot Domain Adaptation for Object Detection via Unified Prompt and Representation Enhancement](https://mlsanity.com/paper/2507.00721/) (2025)
- [An Explainable Deep Neural Network with Frequency-Aware Channel and Spatial Refinement for Flood Prediction in Sustainable Cities](https://mlsanity.com/paper/2509.08003/) (2025)
- [ClassWise-CRF: Category-Specific Fusion for Enhanced Semantic Segmentation of Remote Sensing Imagery](https://mlsanity.com/paper/2504.21491/) (2025)
- [Mitigating Spurious Correlations in Weakly Supervised Semantic Segmentation via Cross-architecture Consistency Regularization](https://mlsanity.com/paper/2507.21959/) (2025)
- [UI-Styler: Ultrasound Image Style Transfer with Class-Aware Prompts for Cross-Device Diagnosis Using a Frozen Black-Box Inference Network](https://mlsanity.com/paper/2511.17155/) (2025)
- [Learning from Heterogeneous Structural MRI via Collaborative Domain Adaptation for Late-Life Depression Assessment](https://mlsanity.com/paper/2507.22321/) (2025)
- [DNA: Dual-branch Network with Adaptation for Open-Set Online Handwriting Generation](https://mlsanity.com/paper/2511.22064/) (2025)
- [Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AI](https://mlsanity.com/paper/2608.05258/) (2026)
- [CoViT: Instance-Correspondence Contrastive Learning for Vision Transformer](https://mlsanity.com/paper/2609.01787/) (2026)
- [Enhancing Shrimp Disease Detection via Deep Learning and Data Refinement for Resilient Aquaculture](https://mlsanity.com/paper/2609.23397/) (2026)

## Abstract (arXiv)

Most domain adaptation (DA) methods are based on either a convolutional neural networks (CNNs) or a vision transformers (ViTs). They align the distribution differences between domains as encoders without considering their unique characteristics. For instance, ViT excels in accuracy due to its superior ability to capture global representations, while CNN has an advantage in capturing local representations. This fact has led us to design a hybrid method to fully take advantage of both ViT and CNN, called Explicitly Class-specific Boundaries (ECB). ECB learns CNN on ViT to combine their distinct strengths. In particular, we leverage ViT's properties to explicitly find class-specific decision boundaries by maximizing the discrepancy between the outputs of the two classifiers to detect target samples far from the source support. In contrast, the CNN encoder clusters target features based on the previously defined class-specific boundaries by minimizing the discrepancy between the probabilities of the two classifiers. Finally, ViT and CNN mutually exchange knowledge to improve the quality of pseudo labels and reduce the knowledge discrepancies of these models. Compared to conventional DA methods, our ECB achieves superior performance, which verifies its effectiveness in this hybrid model. The project website can be found https://dotrannhattuong.github.io/ECB/website.