# Software-Hardware Co-optimization for Modular E2E AV Paradigm: A Unified Framework of Optimization Approaches, Simulation Environment and Evaluation Metrics

arXiv: https://arxiv.org/abs/2601.07393 · HTML: https://mlsanity.com/paper/2601.07393/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning](https://mlsanity.com/paper/2207.07601/) (2022)
- [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024)
- [Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving](https://mlsanity.com/paper/2406.03877/) (2024)
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [Parting with Misconceptions about Learning-based Vehicle Motion Planning](https://mlsanity.com/paper/2306.07962/) (2023)
- [Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?](https://mlsanity.com/paper/2312.03031/) (2023)
- [SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation](https://mlsanity.com/paper/2405.19620/) (2024)
- [GenAD: Generative End-to-End Autonomous Driving](https://mlsanity.com/paper/2402.11502/) (2024)
- [Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes](https://mlsanity.com/paper/2305.10430/) (2023)
- [Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving](https://mlsanity.com/paper/2305.06242/) (2023)
- [Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research](https://mlsanity.com/paper/2310.08710/) (2023)
- [Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations](https://mlsanity.com/paper/2008.05930/) (2020)
- [Learning from All Vehicles](https://mlsanity.com/paper/2203.11934/) (2022)
- [DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.07656/) (2025)
- [PlanT: Explainable Planning Transformers via Object-Level Representations](https://mlsanity.com/paper/2210.14222/) (2022)
- [Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving](https://mlsanity.com/paper/2403.04133/) (2024)
- [HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Driving](https://mlsanity.com/paper/2401.03160/) (2024)
- [HUTFormer: Hierarchical U-Net Transformer for Long-Term Traffic Forecasting](https://mlsanity.com/paper/2307.14596/) (2023)
- [Car-STAGE: Automated framework for large-scale high-dimensional simulated time-series data generation based on user-defined criteria](https://mlsanity.com/paper/2503.03100/) (2025)

## Abstract (arXiv)

Modular end-to-end (ME2E) autonomous driving paradigms combine modular interpretability with global optimization capability and have demonstrated strong performance. However, existing studies mainly focus on accuracy improvement, while critical system-level factors such as inference latency and energy consumption are often overlooked, resulting in increasingly complex model designs that hinder practical deployment. Prior efforts on model compression and acceleration typically optimize either the software or hardware side in isolation. Software-only optimization cannot fundamentally remove intermediate tensor access and operator scheduling overheads, whereas hardware-only optimization is constrained by model structure and precision. As a result, the real-world benefits of such optimizations are often limited. To address these challenges, this paper proposes a reusable software and hardware co-optimization and closed-loop evaluation framework for ME2E autonomous driving inference. The framework jointly integrates software-level model optimization with hardware-level computation optimization under a unified system-level objective. In addition, a multidimensional evaluation metric is introduced to assess system performance by jointly considering safety, comfort, efficiency, latency, and energy, enabling quantitative comparison of different optimization strategies. Experiments across multiple ME2E autonomous driving stacks show that the proposed framework preserves baseline-level driving performance while significantly reducing inference latency and energy consumption, achieving substantial overall system-level improvements. These results demonstrate that the proposed framework provides practical and actionable guidance for efficient deployment of ME2E autonomous driving systems.