# Multi Self-supervised Pre-fine-tuned Transformer Fusion for Better Intelligent Transportation Detection

arXiv: https://arxiv.org/abs/2310.11307 · HTML: https://mlsanity.com/paper/2310.11307/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://mlsanity.com/paper/2201.09450/) (2022)
- [Self-Supervised Learning with Swin Transformers](https://mlsanity.com/paper/2105.04553/) (2021)
- [Attentional Feature Fusion](https://mlsanity.com/paper/2009.14082/) (2020)
- [Ensembling Off-the-shelf Models for GAN Training](https://mlsanity.com/paper/2112.09130/) (2021)
- [Clover: Towards A Unified Video-Language Alignment and Fusion Model](https://mlsanity.com/paper/2207.07885/) (2022)
- [Cross-Domain Car Detection Using Unsupervised Image-to-Image Translation: From Day to Night](https://mlsanity.com/paper/1907.08719/) (2019)
- [An Efficient and Scalable Deep Learning Approach for Road Damage Detection](https://mlsanity.com/paper/2011.09577/) (2020)
- [P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning](https://mlsanity.com/paper/2205.01886/) (2022)
- [Boosting Out-of-Distribution Detection with Multiple Pre-trained Models](https://mlsanity.com/paper/2212.12720/) (2022)
- [BL-MNE: Emerging Heterogeneous Social Network Embedding through Broad Learning with Aligned Autoencoder](https://mlsanity.com/paper/1711.09409/) (2017)
- [BL-ECD: Broad Learning based Enterprise Community Detection via Hierarchical Structure Fusion](https://mlsanity.com/paper/1711.09411/) (2017)

## Abstract (arXiv)

Intelligent transportation system combines advanced information technology to provide intelligent services such as monitoring, detection, and early warning for modern transportation. Intelligent transportation detection is the cornerstone of many intelligent traffic services by identifying task targets through object detection methods. However existing detection methods in intelligent transportation are limited by two aspects. First, there is a difference between the model knowledge pre-trained on large-scale datasets and the knowledge required for target task. Second, most detection models follow the pattern of single-source learning, which limits the learning ability. To address these problems, we propose a Multi Self-supervised Pre-fine-tuned Transformer Fusion (MSPTF) network, consisting of two steps: unsupervised pre-fine-tune domain knowledge learning and multi-model fusion target task learning. In the first step, we introduced self-supervised learning methods into transformer model pre-fine-tune which could reduce data costs and alleviate the knowledge gap between pre-trained model and target task. In the second step, we take feature information differences between different model architectures and different pre-fine-tune tasks into account and propose Multi-model Semantic Consistency Cross-attention Fusion (MSCCF) network to combine different transformer model features by considering channel semantic consistency and feature vector semantic consistency, which obtain more complete and proper fusion features for detection task. We experimented the proposed method on vehicle recognition dataset and road disease detection dataset and achieved 1.1%, 5.5%, 4.2% improvement compared with baseline and 0.7%, 1.8%, 1.7% compared with sota, which proved the effectiveness of our method.