mlsanityData to 14 Oct 2026

AI Act Evaluation Benchmark: An Open, Transparent, and Reproducible Evaluation Dataset for NLP and RAG Systems

arXiv 2603.09435 ↗First posted 10 Mar 2026cs.AIcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

The rapid rollout of AI in heterogeneous public and societal sectors has subsequently escalated the need for compliance with regulatory standards and frameworks. The EU AI Act has emerged as a landmark in the regulatory landscape. The development of solutions that elicit the level of AI systems' compliance with such standards is often limited by the lack of resources, hindering the semi-automated or automated evaluation of their performance. This generates the need for manual work, which is often error-prone, resource-limited or limited to cases not clearly described by the regulation. This paper presents an open, transparent, and reproducible method of creating a resource that facilitates the evaluation of NLP models with a strong focus on RAG systems. We have developed a dataset that contain the tasks of risk-level classification, article retrieval, obligation generation, and question-answering for the EU AI Act. The dataset files are in a machine-to-machine appropriate format. To generate the files, we utilise domain knowledge as an exegetical basis, combining with the processing and reasoning power of large language models to generate scenarios along with the respective tasks. Our methodology demonstrates a way to harness language models for grounded generation with high document relevancy. Besides, we overcome limitations such as navigating the decision boundaries of risk-levels that are not explicitly defined within the EU AI Act, such as limited and minimal cases. Finally, we demonstrate our dataset's effectiveness by evaluating a RAG-based solution that reaches 0.87 and 0.85 F1-score for prohibited and high-risk scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 5 of its 18 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

5 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 4

  1. 2024COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Actcited by 24
  2. 2025Mapping the Regulatory Learning Space for the EU AI Actcited by 3
  3. 2025AI for Regulatory Affairs: Balancing Accuracy, Interpretability, and Computational Cost in Medical Device Classificationcited by 3
  4. 2025Navigating the EU AI Act: Foreseeable Challenges in Qualifying Deep Learning-Based Automated Inspections of Class III Medical Devicescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The illusion of a perfect metric: Why evaluating AI's words is harder than it looks
  2. 2026LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
  3. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  4. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
  5. 2025Towards a rigorous evaluation of RAG systems: the challenge of due diligence
  6. 2026Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
  7. 2025A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent
  8. 2025Knowledge-Graph Based RAG System Evaluation Framework