# CAN We Trust Your Results? A Cross-Dataset Study of Automotive IDS Evaluation

arXiv: https://arxiv.org/abs/2606.30430 · HTML: https://mlsanity.com/paper/2606.30430/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 11 of 11)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [CANet: An Unsupervised Intrusion Detection System for High Dimensional CAN Bus Data](https://mlsanity.com/paper/1906.02492/) (2019)
- [A Comprehensive Guide to CAN IDS Data & Introduction of the ROAD Dataset](https://mlsanity.com/paper/2012.14600/) (2020)
- [INDRA: Intrusion Detection using Recurrent Autoencoders in Automotive Embedded Systems](https://mlsanity.com/paper/2007.08795/) (2020)
- [CAN-D: A Modular Four-Step Pipeline for Comprehensively Decoding Controller Area Network Data](https://mlsanity.com/paper/2006.05993/) (2020)
- [CANShield: Deep Learning-Based Intrusion Detection Framework for Controller Area Networks at the Signal-Level](https://mlsanity.com/paper/2205.01306/) (2022)
- [Cyberattacks and Countermeasures For In-Vehicle Networks](https://mlsanity.com/paper/2004.10781/) (2020)
- [State-of-the-Art Survey on In-Vehicle Network Communication (CAN-Bus) Security and Vulnerabilities](https://mlsanity.com/paper/1802.01725/) (2018)
- [Time-Based CAN Intrusion Detection Benchmark](https://mlsanity.com/paper/2101.05781/) (2021)
- [Performance comparison of timing-based anomaly detectors for Controller Area Network: a reproducible study](https://mlsanity.com/paper/2307.04561/) (2023)
- [Benchmarking Frameworks and Comparative Studies of Controller Area Network (CAN) Intrusion Detection Systems: A Review](https://mlsanity.com/paper/2402.06904/) (2024)

## Abstract (arXiv)

The increasing connectivity of modern vehicles has made securing in-vehicle communication networks a critical challenge. Intrusion Detection Systems (IDS) have been widely studied as a defense mechanism for detecting malicious activities on the Controller Area Network (CAN) bus. However, the evaluation of CAN IDS methods remains difficult due to inconsistencies in experimental setups and the lack of standardized benchmarking frameworks. As a result, reported performance often depends on dataset-specific characteristics and may not reflect how detection methods behave in different environments. This work introduces a benchmarking framework for consistent evaluation of CAN IDSs across multiple datasets. Using the proposed framework, we integrate seven publicly available CAN IDS datasets collected under different experimental conditions and perform cross-dataset evaluation of five conceptually different IDS approaches. Our results highlight how detection performance can vary significantly across datasets, demonstrating the importance of cross-dataset benchmarking for assessing the robustness and generalization capabilities of CAN IDS methods.