mlsanityData to 14 Oct 2026

HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning

arXiv 2511.15355 ↗First posted 19 Nov 2025cs.CLcited by 1

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

We introduce HEAD-QA v2, an expanded and updated version of a Spanish/English healthcare multiple-choice reasoning dataset originally released by Vilares and G'omez-Rodr'iguez (2019). The update responds to the growing need for high-quality datasets that capture the linguistic and conceptual complexity of healthcare reasoning. We extend the dataset to over 12,000 questions from ten years of Spanish professional exams, benchmark several open-source LLMs using prompting, RAG, and probability-based answer selection, and provide additional multilingual versions to support future work. Results indicate that performance is mainly driven by model scale and intrinsic reasoning ability, with complex inference strategies obtaining limited gains. Together, these results establish HEAD-QA v2 as a reliable resource for advancing research on biomedical reasoning and model improvement.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 26 of its 34 references that are in the map. First posted on arXiv 19 Nov 2025.

Builds on

26 of 34 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  9. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  10. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  11. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  12. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
14 more
  1. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  2. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  3. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 185
  4. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 145
  5. 2023What's In My Big Data?cited by 129
  6. 2015Ask Me Anything: Dynamic Memory Networks for Natural Language Processingcited by 117
  7. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 101
  8. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 90
  9. 2025LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?cited by 55
  10. 2025Learning without training: The implicit dynamics of in-context learningcited by 43
  11. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
  12. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
  13. 2023FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domaincited by 25
  14. 2025BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translationcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Taskscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Disentangling Reasoning and Knowledge in Medical Large Language Models
  2. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
  3. 2025MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation
  4. 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine
  5. 2026Medical Reasoning with Large Language Models: A Survey and MR-Bench
  6. 2025Multilingual Medical Reasoning for Question Answering with Large Language Models
  7. 2024Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
  8. 2025Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications