# Improving Generalization of Transfer Learning Across Domains Using Spatio-Temporal Features in Autonomous Driving

arXiv: https://arxiv.org/abs/2103.08116 · HTML: https://mlsanity.com/paper/2103.08116/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 15 Mar 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [A2D2: Audi Autonomous Driving Dataset](https://mlsanity.com/paper/2004.06320/) (2020)
- [Image to Image Translation for Domain Adaptation](https://mlsanity.com/paper/1712.00479/) (2017)
- [End-to-end Learning of Driving Models from Large-scale Video Datasets](https://mlsanity.com/paper/1612.01079/) (2016)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning](https://mlsanity.com/paper/1811.02307/) (2018)
- [Canadian Adverse Driving Conditions Dataset](https://mlsanity.com/paper/2001.10117/) (2020)
- [DDD17: End-To-End DAVIS Driving Dataset](https://mlsanity.com/paper/1711.01458/) (2017)
- [Virtual to Real Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1704.03952/) (2017)
- [Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks](https://mlsanity.com/paper/1707.05740/) (2017)
- [Temporal-Channel Transformer for 3D Lidar-Based Video Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.13628/) (2020)
- [Looking at the right stuff: Guided semantic-gaze for autonomous driving](https://mlsanity.com/paper/1911.10455/) (2019)
- [End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies](https://mlsanity.com/paper/1710.03804/) (2017)
- [Enhanced Transfer Learning for Autonomous Driving with Systematic Accident Simulation](https://mlsanity.com/paper/2007.12148/) (2020)
- [Virtual-to-Real-World Transfer Learning for Robots on Wilderness Trails](https://mlsanity.com/paper/1901.05599/) (2019)
- [Deep Learning Based Motion Planning For Autonomous Vehicle Using Spatiotemporal LSTM Network](https://mlsanity.com/paper/1903.01712/) (2019)

## Built on by (most cited first, 5 of 5)

- [RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding](https://mlsanity.com/paper/2304.08600/) (2023)
- [Softmax-free Linear Transformers](https://mlsanity.com/paper/2207.03341/) (2022)
- [STLight: a Fully Convolutional Approach for Efficient Predictive Learning by Spatio-Temporal joint Processing](https://mlsanity.com/paper/2411.10198/) (2024)
- [Learning Isometric Embeddings of Road Networks using Multidimensional Scaling](https://mlsanity.com/paper/2504.17534/) (2025)
- [A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platform](https://mlsanity.com/paper/2610.00926/) (2026)

## Abstract (arXiv)

Practical learning-based autonomous driving models must be capable of generalizing learned behaviors from simulated to real domains, and from training data to unseen domains with unusual image properties. In this paper, we investigate transfer learning methods that achieve robustness to domain shifts by taking advantage of the invariance of spatio-temporal features across domains. In this paper, we propose a transfer learning method to improve generalization across domains via transfer of spatio-temporal features and salient data augmentation. Our model uses a CNN-LSTM network with Inception modules for image feature extraction. Our method runs in two phases: Phase 1 involves training on source domain data, while Phase 2 performs training on target domain data that has been supplemented by feature maps generated using the Phase 1 model. Our model significantly improves performance in unseen test cases for both simulation-to-simulation transfer as well as simulation-to-real transfer by up to +37.3% in test accuracy and up to +40.8% in steering angle prediction, compared to other SOTA methods across multiple datasets.