# QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation

arXiv: https://arxiv.org/abs/2505.05225 · HTML: https://mlsanity.com/paper/2505.05225/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 46 references that are in the map.
- First posted on arXiv 8 May 2025.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](https://mlsanity.com/paper/2305.12474/) (2023)
- [FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning](https://mlsanity.com/paper/2309.00363/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Rethinking LLM Memorization through the Lens of Adversarial Compression](https://mlsanity.com/paper/2404.15146/) (2024)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning](https://mlsanity.com/paper/2310.15205/) (2023)
- [FinGPT: Large Generative Models for a Small Language](https://mlsanity.com/paper/2311.05640/) (2023)
- [Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements](https://mlsanity.com/paper/2502.12904/) (2025)
- [Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset](https://mlsanity.com/paper/2405.10542/) (2024)
- [Bridge the Domains: Large Language Models Enhanced Cross-domain Sequential Recommendation](https://mlsanity.com/paper/2504.18383/) (2025)
- [Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialogues](https://mlsanity.com/paper/2412.09049/) (2024)
- [GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation](https://mlsanity.com/paper/2402.15745/) (2024)
- [Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements](https://mlsanity.com/paper/2502.12459/) (2025)

## Built on by (most cited first, 5 of 5)

- [Understanding How Value Neurons Shape the Generation of Specified Values in LLMs](https://mlsanity.com/paper/2505.17712/) (2025)
- [RAL2M: Retrieval Augmented Learning-To-Match Against Hallucination in Compliance-Guaranteed Service Systems](https://mlsanity.com/paper/2601.02917/) (2026)
- [Contextualized Token Discrimination for Speech Search Query Correction](https://mlsanity.com/paper/2509.04393/) (2025)
- [Proactive Dialogue Model with Intent Prediction](https://mlsanity.com/paper/2604.27379/) (2026)
- [Meta-Moderator: Empowering Multi-Agent Debate with Meta-Cognition](https://mlsanity.com/paper/2608.23029/) (2026)

## Abstract (arXiv)

The rapid advancement of Chinese LLMs underscores the need for vertical-domain evaluations to ensure reliable applications. However, existing benchmarks often lack domain coverage and provide limited insights into the Chinese working context. Leveraging qualification exams as a unified framework for expertise evaluation, we introduce QualBench, the first multi-domain Chinese QA benchmark dedicated to localized assessment of Chinese LLMs. The dataset includes over 17,000 questions across six vertical domains, drawn from 24 Chinese qualifications to align with national policies and professional standards. Results reveal an interesting pattern of Chinese LLMs consistently surpassing non-Chinese models, with the Qwen2.5 model outperforming the more advanced GPT-4o, emphasizing the value of localized domain knowledge in meeting qualification requirements. The average accuracy of 53.98% reveals the current gaps in domain coverage within model capabilities. Furthermore, we identify performance degradation caused by LLM crowdsourcing, assess data contamination, and illustrate the effectiveness of prompt engineering and model fine-tuning, suggesting opportunities for future improvements through multi-domain RAG and Federated Learning.