# New Spiking Architecture for Multi-Modal Decision-Making in Autonomous Vehicles

arXiv: https://arxiv.org/abs/2512.01882 · HTML: https://mlsanity.com/paper/2512.01882/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 38 of its 60 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)
- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Deep Residual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2102.04159/) (2021)
- [Spikformer: When Spiking Neural Network Meets Transformer](https://mlsanity.com/paper/2209.15425/) (2022)
- [Training Spiking Neural Networks Using Lessons From Deep Learning](https://mlsanity.com/paper/2109.12894/) (2021)
- [Going Deeper in Spiking Neural Networks: VGG and Residual Architectures](https://mlsanity.com/paper/1802.02627/) (2018)
- [Spike-driven Transformer](https://mlsanity.com/paper/2307.01694/) (2023)
- [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023)
- [Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures](https://mlsanity.com/paper/1903.06379/) (2019)
- [Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review](https://mlsanity.com/paper/2005.09830/) (2020)
- [Multi-modal Sensor Fusion for Auto Driving Perception: A Survey](https://mlsanity.com/paper/2202.02703/) (2022)
- [ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection](https://mlsanity.com/paper/2308.07504/) (2023)
- [Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control](https://mlsanity.com/paper/2010.09635/) (2020)
- [Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation](https://mlsanity.com/paper/2201.07779/) (2022)
- [Optimized Potential Initialization for Low-latency Spiking Neural Networks](https://mlsanity.com/paper/2202.01440/) (2022)
- [Reducing ANN-SNN Conversion Error through Residual Membrane Potential](https://mlsanity.com/paper/2302.02091/) (2023)
- [Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization](https://mlsanity.com/paper/2305.02099/) (2023)
- [RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networks](https://mlsanity.com/paper/2308.06787/) (2023)
- [Human-Level Control through Directly-Trained Deep Spiking Q-Networks](https://mlsanity.com/paper/2201.07211/) (2021)
- [Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence](https://mlsanity.com/paper/2106.01288/) (2021)
- [TCL: an ANN-to-SNN Conversion with Trainable Clipping Layers](https://mlsanity.com/paper/2008.04509/) (2020)
- [High-Performance Temporal Reversible Spiking Neural Networks with O(L) Training Memory and O(1) Inference Cost](https://mlsanity.com/paper/2405.16466/) (2024)
- [Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning](https://mlsanity.com/paper/2107.02339/) (2021)
- [Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformer](https://mlsanity.com/paper/2503.00226/) (2025)

## Built on by (most cited first, 2 of 2)

- [Efficient Aspect Term Extraction using Spiking Neural Network](https://mlsanity.com/paper/2601.06637/) (2026)
- [Fuzzy Encoding-Decoding to Improve Spiking Q-Learning Performance in Autonomous Driving](https://mlsanity.com/paper/2604.16436/) (2026)

## Abstract (arXiv)

This work proposes an end-to-end multi-modal reinforcement learning framework for high-level decision-making in autonomous vehicles. The framework integrates heterogeneous sensory input, including camera images, LiDAR point clouds, and vehicle heading information, through a cross-attention transformer-based perception module. Although transformers have become the backbone of modern multi-modal architectures, their high computational cost limits their deployment in resource-constrained edge environments. To overcome this challenge, we propose a spiking temporal-aware transformer-like architecture that uses ternary spiking neurons for computationally efficient multi-modal fusion. Comprehensive evaluations across multiple tasks in the Highway Environment demonstrate the effectiveness and efficiency of the proposed approach for real-time autonomous decision-making.