mlsanityData to 7 Oct 2026

Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations

arXiv 2303.18027 ↗First posted 31 Mar 2023cs.CLcited by 63

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) gain popularity among speakers of diverse languages, we believe that it is crucial to benchmark them to better understand model behaviors, failures, and limitations in languages beyond English. In this work, we evaluate LLM APIs (ChatGPT, GPT-3, and GPT-4) on the Japanese national medical licensing examinations from the past five years, including the current year. Our team comprises native Japanese-speaking NLP researchers and a practicing cardiologist based in Japan. Our experiments show that GPT-4 outperforms ChatGPT and GPT-3 and passes all six years of the exams, highlighting LLMs' potential in a language that is typologically distant from English. However, our evaluation also exposes critical limitations of the current LLM APIs. First, LLMs sometimes select prohibited choices that should be strictly avoided in medical practice in Japan, such as suggesting euthanasia. Further, our analysis shows that the API costs are generally higher and the maximum context size is smaller for Japanese because of the way non-Latin scripts are currently tokenized in the pipeline. We release our benchmark as Igaku QA as well as all model outputs and exam metadata. We hope that our results and benchmark will spur progress on more diverse applications of LLMs. Our benchmark is available at https://github.com/jungokasai/IgakuQA.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 49 of its 82 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

49 of 82 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
37 more
  1. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  6. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  7. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  8. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  9. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  10. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  11. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  12. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  13. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  14. 2022RealTime QA: What's the Answer Right Now?cited by 239
  15. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  16. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  17. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  18. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  19. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  20. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  21. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  22. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  23. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 96
  24. 2018Neural Cross-Lingual Named Entity Recognition with Minimal Resourcescited by 44
  25. 2021Transparent Human Evaluation for Image Captioningcited by 40
  26. 2019Translationese in Machine Translation Evaluationcited by 38
  27. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  28. 2016Many Languages, One Parsercited by 26
  29. 2021GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generationcited by 20
  30. 2019Polyglot Contextual Representations Improve Crosslingual Transfercited by 16
  31. 2021Fine-Tuning Transformers: Vocabulary Transfercited by 14
  32. 2022MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languagescited by 11
  33. 2020Pre-training technique to localize medical BERT and enhance biomedical BERTcited by 6
  34. 2016Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learningcited by 5
  35. 2018Polyglot Semantic Role Labelingcited by 5
  36. 2019Low-Resource Parsing with Crosslingual Contextualized Representationscited by 5
  37. 2022Cross-Lingual Knowledge Transfer for Clinical Phenotypingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 19; 2025: 18; 2026: 7.

63 papers.

Sort:

By area:

  1. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  2. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  3. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  4. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  5. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  6. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  7. 2024Towards Building Multilingual Language Model for Medicinecited by 91
  8. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  9. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  10. 2025An Agentic System for Rare Disease Diagnosis with Traceable Reasoningcited by 50
  11. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47
  12. 2023Augmented Large Language Models with Parametric Knowledge Guidingcited by 37
38 more
  1. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  2. 2023Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AIcited by 33
  3. 2024ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?cited by 31
  4. 2023Sensitivity and Robustness of Large Language Models to Prompt Template in Japanese Text Classification Taskscited by 25
  5. 2024From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyondcited by 24
  6. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
  7. 2023Decoding ChatGPT: A Taxonomy of Existing Research, Current Challenges, and Possible Future Directionscited by 18
  8. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
  9. 2023Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not todaycited by 14
  10. 2024Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPTcited by 11
  11. 2023Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundariescited by 10
  12. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  13. 2024JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Modelscited by 8
  14. 2024Polish-English medical knowledge transfer: A new benchmark and resultscited by 8
  15. 2023JMedLoRA:Medical Domain Adaptation on Japanese Large Language Models using Instruction-tuningcited by 6
  16. 2023FDAPT: Federated Domain-adaptive Pre-training for Language Modelscited by 5
  17. 2025Benchmarking GPT-5 for Zero-Shot Multimodal Medical Reasoning in Radiology and Radiation Oncologycited by 5
  18. 2023Evaluating multiple large language models in pediatric ophthalmologycited by 4
  19. 2025Teaching AI to Handle Exceptions: Supervised Fine-Tuning with Human-Aligned Judgmentcited by 4
  20. 2025Stabilizing Reasoning in Medical LLMs with Continued Pretraining and Reasoning Preference Optimizationcited by 4
  21. 2024AutoRD: An Automatic and End-to-End System for Rare Disease Knowledge Graph Construction Based on Ontologies-enhanced Large Language Modelscited by 3
  22. 202470B-parameter large language models in Japanese medical question-answeringcited by 3
  23. 2024Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resourcescited by 3
  24. 2025LocalGPT: Benchmarking and Advancing Large Language Models for Local Life Services in Meituancited by 3
  25. 2025KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinationscited by 3
  26. 2025CoT-X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimizationcited by 3
  27. 2026ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Makingcited by 3
  28. 2024Are Large Language Models True Healthcare Jacks-of-All-Trades? Benchmarking Across Health Professions Beyond Physician Examscited by 2
  29. 2024Effectiveness of ChatGPT in explaining complex medical reports to patientscited by 2
  30. 2025Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgmentscited by 2
  31. 2025A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLPcited by 2
  32. 2025MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Textscited by 2
  33. 2026Med-CoReasoner: Reducing Language Disparities in Medical Reasoning via Language-Informed Co-Reasoningcited by 2
  34. 2024A Tutorial on Clinical Speech AI Development: From Data Collection to Model Validationcited by 1
  35. 2025Analyzing the Safety of Japanese Large Language Models in Stereotype-Triggering Promptscited by 1
  36. 2025Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutionscited by 1
  37. 2025Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integrationcited by 1
  38. 2025Filling in the Clinical Gaps in Benchmark: Case for HealthBench for the Japanese medical systemcited by 1

50 most cited of 63 All 63 (JSON)