# Lightweight Temporal Transformer Decomposition for Federated Autonomous Driving

arXiv: https://arxiv.org/abs/2506.23523 · HTML: https://mlsanity.com/paper/2506.23523/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 69 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 30 of 35)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [Tensor Decomposition for Signal Processing and Machine Learning](https://mlsanity.com/paper/1607.01668/) (2016)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms](https://mlsanity.com/paper/1808.07576/) (2018)
- [ReasonNet: End-to-End Driving with Temporal and Global Reasoning](https://mlsanity.com/paper/2305.10507/) (2023)
- [Federated Learning Meets Blockchain in Edge Computing: Opportunities and Challenges](https://mlsanity.com/paper/2104.01776/) (2021)
- [MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling](https://mlsanity.com/paper/1905.09435/) (2019)
- [Deep Federated Learning for Autonomous Driving](https://mlsanity.com/paper/2110.05754/) (2021)
- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [Federated Transfer Reinforcement Learning for Autonomous Driving](https://mlsanity.com/paper/1910.06001/) (2019)
- [Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges](https://mlsanity.com/paper/2308.10407/) (2023)
- [Throughput-Optimal Topology Design for Cross-Silo Federated Learning](https://mlsanity.com/paper/2010.12229/) (2020)
- [Pseudo-Stereo for Monocular 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/2203.02112/) (2022)
- [Investigating the Impact of Multi-LiDAR Placement on Object Detection for Autonomous Driving](https://mlsanity.com/paper/2105.00373/) (2021)
- [Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs](https://mlsanity.com/paper/2008.09342/) (2020)
- [Practical Privacy Preserving POI Recommendation](https://mlsanity.com/paper/2003.02834/) (2020)
- [Trajectory Distribution Control for Model Predictive Path Integral Control using Covariance Steering](https://mlsanity.com/paper/2109.12147/) (2021)
- [Online Spatio-Temporal Correlation-Based Federated Learning for Traffic Flow Forecasting](https://mlsanity.com/paper/2302.08658/) (2023)
- [BFRT: Blockchained Federated Learning for Real-time Traffic Flow Prediction](https://mlsanity.com/paper/2305.17677/) (2023)
- [Robust Tensor Decomposition for Image Representation Based on Generalized Correntropy](https://mlsanity.com/paper/2005.04605/) (2020)
- [PEPPER: Empowering User-Centric Recommender Systems over Gossip Learning](https://mlsanity.com/paper/2208.05320/) (2022)

## Abstract (arXiv)

Traditional vision-based autonomous driving systems often face difficulties in navigating complex environments when relying solely on single-image inputs. To overcome this limitation, incorporating temporal data such as past image frames or steering sequences, has proven effective in enhancing robustness and adaptability in challenging scenarios. While previous high-performance methods exist, they often rely on resource-intensive fusion networks, making them impractical for training and unsuitable for federated learning. To address these challenges, we propose lightweight temporal transformer decomposition, a method that processes sequential image frames and temporal steering data by breaking down large attention maps into smaller matrices. This approach reduces model complexity, enabling efficient weight updates for convergence and real-time predictions while leveraging temporal information to enhance autonomous driving performance. Intensive experiments on three datasets demonstrate that our method outperforms recent approaches by a clear margin while achieving real-time performance. Additionally, real robot experiments further confirm the effectiveness of our method.