# An Empirical Study and Assessment of EU AI Act Compliance Checkers

arXiv: https://arxiv.org/abs/2609.36228 · HTML: https://mlsanity.com/paper/2609.36228/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 59 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act](https://mlsanity.com/paper/2410.07959/) (2024)
- [Conformity Assessments and Post-market Monitoring: A Guide to the Role of Auditing in the Proposed European AI Regulation](https://mlsanity.com/paper/2111.05071/) (2021)
- [Robustness and Cybersecurity in the EU Artificial Intelligence Act](https://mlsanity.com/paper/2502.16184/) (2025)
- [AI Cards: Towards an Applied Framework for Machine-Readable AI and Risk Documentation Inspired by the EU AI Act](https://mlsanity.com/paper/2406.18211/) (2024)
- [Unpacking Privacy Labels: A Measurement and Developer Perspective on Google's Data Safety Section](https://mlsanity.com/paper/2306.08111/) (2023)
- [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025)
- [Is It a Trap? A Large-scale Empirical Study And Comprehensive Assessment of Online Automated Privacy Policy Generators for Mobile Apps](https://mlsanity.com/paper/2305.03271/) (2023)
- [Quality Assessment of Online Automated Privacy Policy Generators: An Empirical Study](https://mlsanity.com/paper/2002.05341/) (2020)
- [False Sense of Security in Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2405.03820/) (2024)
- [Compliance Cards: Automated EU AI Act Compliance Analyses amidst a Complex AI Supply Chain](https://mlsanity.com/paper/2406.14758/) (2024)
- [TechOps: Technical Documentation Templates for the AI Act](https://mlsanity.com/paper/2508.08804/) (2025)
- [ESG Reporting Lifecycle Management with Large Language Models and AI Agents](https://mlsanity.com/paper/2603.10646/) (2026)
- [On the Complexities of Testing for Compliance with Human Oversight Requirements in AI Regulation](https://mlsanity.com/paper/2504.03300/) (2025)
- [Test-takers have a say: understanding the implications of the use of AI in language tests](https://mlsanity.com/paper/2307.09885/) (2023)
- [The Creation and Analysis of Government AI Transparency Statements in Australia](https://mlsanity.com/paper/2604.26075/) (2026)
- [Resume-ing Control: (Mis)Perceptions of Agency Around GenAI Use in Recruiting Workflows](https://mlsanity.com/paper/2604.26851/) (2026)

## Abstract (arXiv)

The EU AI Act introduces extensive compliance requirements for organizations that develop, deploy, or integrate AI systems. Many of these requirements are directly relevant to security and privacy, while also addressing closely related issues such as data governance, transparency, accuracy, and robustness. However, stakeholders such as small-to-medium businesses and individual developers often lack the legal expertise required to interpret these obligations and translate them into engineering and governance practices. This disconnect creates challenges for implementing the EU AI Act and may lead to missing safeguards or misdirected development and deployment efforts. To address this, various automated EU AI Act compliance checkers (AIACCs) have emerged, claiming to streamline compliance assessments and provide practical guidance. In this paper, we present the first empirical study and assessment of AIACCs. We characterize 12 mainstream AIACCs across multiple dimensions, evaluate their legal coverage and alignment, and analyze checker-generated compliance reports for structure, determinacy, and actionability. We find that the quality of AIACCs varies significantly and that they currently can only serve as early-stage orientation tools. Specifically, we observe inconsistent interaction modes and user-friendliness, a tendency to overly simplify or omit key obligations, and a failure to provide determinate, actionable guidance. As a result, reliance on the current generation of AIACCs may foster a false sense of compliance. With our study, we provide a critical baseline of the current AIACC landscape. We further offer design principles for the implementation of more reliable compliance-support tools.