# Automated Benchmark Generation from Domain Guidelines Informed by Bloom's Taxonomy

arXiv: https://arxiv.org/abs/2601.20253 · HTML: https://mlsanity.com/paper/2601.20253/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 36 references that are in the map.
- First posted on arXiv 28 Jan 2026.

## Builds on (most cited first, 12 of 12)

- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks](https://mlsanity.com/paper/2410.20266/) (2024)
- [Position: AI Evaluation Should Learn from How We Test Humans](https://mlsanity.com/paper/2306.10512/) (2023)
- [MOPI-HFRS: A Multi-objective Personalized Health-aware Food Recommendation System with LLM-enhanced Interpretation](https://mlsanity.com/paper/2412.08847/) (2024)
- [NGQA: A Nutritional Graph Question Answering Benchmark for Personalized Health-aware Nutritional Reasoning](https://mlsanity.com/paper/2412.15547/) (2024)
- [FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination](https://mlsanity.com/paper/2406.10261/) (2024)
- [DischargeSim: A Simulation Benchmark for Educational Doctor-Patient Communication at Discharge](https://mlsanity.com/paper/2509.07188/) (2025)
- [BloomAPR: A Bloom's Taxonomy-based Framework for Assessing the Capabilities of LLM-Powered APR Solutions](https://mlsanity.com/paper/2509.25465/) (2025)
- [QG-SMS: Enhancing Test Item Analysis via Student Modeling and Simulation](https://mlsanity.com/paper/2503.05888/) (2025)
- [Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving](https://mlsanity.com/paper/2506.08349/) (2025)
- [Building Scaffolding Dialogue Data with LLM-Simulated Novices](https://mlsanity.com/paper/2508.04428/) (2025)

## Built on by (most cited first, 4 of 4)

- [Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models](https://mlsanity.com/paper/2606.05531/) (2026)
- [CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models](https://mlsanity.com/paper/2604.05755/) (2026)
- [MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models](https://mlsanity.com/paper/2609.38543/) (2026)
- [ChartBmkAgent: Harness-Governed Multi-Agent Construction of Chart QA Benchmarks from Sparse Error-Taxonomy Specifications](https://mlsanity.com/paper/2610.08106/) (2026)

## Abstract (arXiv)

Open-ended question answering (QA) evaluates a model's ability to perform contextualized reasoning beyond factual recall. This challenge is especially acute in practice-based domains, where knowledge is procedural and grounded in professional judgment, while most existing LLM benchmarks depend on pre-existing human exam datasets that are often unavailable in such settings. We introduce a framework for automated benchmark generation from expert-authored guidelines informed by Bloom's Taxonomy. It converts expert practices into implicit violation-based scenarios and expands them into auto-graded multiple-choice questions (MCQs) and multi-turn dialogues across four cognitive levels, enabling deterministic, reproducible, and scalable evaluation. Applied to three applied domains: teaching, dietetics, and caregiving, we find differences between model and human-like reasoning: LLMs sometimes perform relatively better on higher-order reasoning (Analyze) but fail more frequently on lower-level items (Remember). We produce large-scale, psychometrically informed benchmarks that surface these non-intuitive model behaviors and enable evaluation of contextualized reasoning in real-world settings.