# JobResQA: A Benchmark for LLM Machine Reading Comprehension on Multilingual R'esum'es and JDs

arXiv: https://arxiv.org/abs/2601.23183 · HTML: https://mlsanity.com/paper/2601.23183/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 17 of 17)

- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2309.11925/) (2023)
- [Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering](https://mlsanity.com/paper/2210.01613/) (2022)
- [Complex Knowledge Base Question Answering: A Survey](https://mlsanity.com/paper/2108.06688/) (2021)
- [SkillSpan: Hard and Soft Skill Extraction from English Job Postings](https://mlsanity.com/paper/2204.12811/) (2022)
- [TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinement](https://mlsanity.com/paper/2402.16379/) (2024)
- [Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study](https://mlsanity.com/paper/2502.02481/) (2025)
- ["You Gotta be a Doctor, Lin": An Investigation of Name-Based Bias of Large Language Models in Employment Recommendations](https://mlsanity.com/paper/2406.12232/) (2024)
- [HR-MultiWOZ: A Task Oriented Dialogue (TOD) Dataset for HR LLM Agent](https://mlsanity.com/paper/2402.01018/) (2024)
- [LLMs Are Zero-Shot Context-Aware Simultaneous Translators](https://mlsanity.com/paper/2406.13476/) (2024)
- [Natural Language Processing for Human Resources: A Survey](https://mlsanity.com/paper/2410.16498/) (2024)
- [Invisible Filters: Cultural Bias in Hiring Evaluations Using Large Language Models](https://mlsanity.com/paper/2508.16673/) (2025)
- [ConFit v2: Improving Resume-Job Matching using Hypothetical Resume Embedding and Runner-Up Hard-Negative Mining](https://mlsanity.com/paper/2502.12361/) (2025)
- [R'esum'e Parsing as Hierarchical Sequence Labeling: An Empirical Study](https://mlsanity.com/paper/2309.07015/) (2023)
- [Synthetic CVs To Build and Test Fairness-Aware Hiring Tools](https://mlsanity.com/paper/2508.21179/) (2025)

## Built on by (most cited first, 2 of 2)

- [Overview of the TalentCLEF 2026: Skill and Job Title Intelligence for Human Capital Management](https://mlsanity.com/paper/2606.31692/) (2026)
- [From Matching Models to Recruiting Agents: A Systematized Narrative Review of AI Recruitment Systems, Evaluation, and Governance](https://mlsanity.com/paper/2609.04286/) (2026)

## Abstract (arXiv)

We introduce JobResQA, a multilingual Question Answering benchmark for evaluating Machine Reading Comprehension (MRC) capabilities of LLMs on HR-specific tasks involving r'esum'es and job descriptions. The dataset comprises 581 QA pairs across 105 synthetic r'esum'e-job description pairs in five languages (English, Spanish, Italian, German, and Chinese), with questions spanning three complexity levels from basic factual extraction to complex cross-document reasoning. We propose a data generation pipeline derived from real-world sources through de-identification and data synthesis to ensure both realism and privacy, while controlled demographic and professional attributes (implemented via placeholders) enable systematic bias and fairness studies. We also present a cost-effective, human-in-the-loop translation pipeline based on the TEaR methodology, incorporating MQM error annotations and selective post-editing to ensure an high-quality multi-way parallel benchmark. We provide a baseline evaluations across multiple open-weight LLM families using an LLM-as-judge approach revealing higher performances on English and Spanish but substantial degradation for other languages, highlighting critical gaps in multilingual MRC capabilities for HR applications. JobResQA provides a reproducible benchmark for advancing fair and reliable LLM-based HR systems. The benchmark is publicly available at: https://github.com/Avature/jobresqa-benchmark