# ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain

arXiv: https://arxiv.org/abs/2305.12092 · HTML: https://mlsanity.com/paper/2305.12092/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 68 references that are in the map.
- First posted on arXiv 20 May 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices](https://mlsanity.com/paper/1906.09208/) (2019)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [Rethinking embedding coupling in pre-trained language models](https://mlsanity.com/paper/2010.12821/) (2020)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [SkillSpan: Hard and Soft Skill Extraction from English Job Postings](https://mlsanity.com/paper/2204.12811/) (2022)
- [What does it mean to solve the problem of discrimination in hiring? Social, technical and legal perspectives from the UK on automated hiring systems](https://mlsanity.com/paper/1910.06144/) (2019)
- [Dict-BERT: Enhancing Language Model Pre-training with Dictionary](https://mlsanity.com/paper/2110.06490/) (2021)
- [JobBERT: Understanding Job Titles through Skills](https://mlsanity.com/paper/2109.09605/) (2021)
- [Learning Representations for Soft Skill Matching](https://mlsanity.com/paper/1807.07741/) (2018)
- [Design of Negative Sampling Strategies for Distantly Supervised Skill Extraction](https://mlsanity.com/paper/2209.05987/) (2022)
- [DataOps for Societal Intelligence: a Data Pipeline for Labor Market Skills Extraction and Matching](https://mlsanity.com/paper/2104.01966/) (2021)
- [Kompetencer: Fine-grained Skill Classification in Danish Job Postings via Distant Supervision and Transfer Learning](https://mlsanity.com/paper/2205.01381/) (2022)
- [Skill Extraction from Job Postings using Weak Supervision](https://mlsanity.com/paper/2209.08071/) (2022)
- [Salience and Market-aware Skill Extraction for Job Targeting](https://mlsanity.com/paper/2005.13094/) (2020)

## Built on by (most cited first, 14 of 14)

- [Extreme Multi-Label Skill Extraction Training using Large Language Models](https://mlsanity.com/paper/2307.10778/) (2023)
- [Large Language Models as Batteries-Included Zero-Shot ESCO Skills Matchers](https://mlsanity.com/paper/2307.03539/) (2023)
- [Deep Learning-based Computational Job Market Analysis: A Survey on Skill Extraction and Classification from Job Postings](https://mlsanity.com/paper/2402.05617/) (2024)
- [Rethinking Skill Extraction in the Job Market Domain using Large Language Models](https://mlsanity.com/paper/2402.03832/) (2024)
- [Natural Language Processing for Human Resources: A Survey](https://mlsanity.com/paper/2410.16498/) (2024)
- [Skill-LLM: Repurposing General-Purpose LLMs for Skill Extraction](https://mlsanity.com/paper/2410.12052/) (2024)
- [MELO: An Evaluation Benchmark for Multilingual Entity Linking of Occupations](https://mlsanity.com/paper/2410.08319/) (2024)
- [NNOSE: Nearest Neighbor Occupational Skill Extraction](https://mlsanity.com/paper/2401.17092/) (2024)
- [CareerBERT: Matching Resumes to ESCO Jobs in a Shared Embedding Space for Generic Job Recommendations](https://mlsanity.com/paper/2503.02056/) (2025)
- [Overview of the TalentCLEF 2025: Skill and Job Title Intelligence for Human Capital Management](https://mlsanity.com/paper/2507.13275/) (2025)
- [Enhancing PLM Performance on Labour Market Tasks via Instruction-based Finetuning and Prompt-tuning with Rules](https://mlsanity.com/paper/2308.16770/) (2023)
- [NLPnorth @ TalentCLEF 2025: Comparing Discriminative, Contrastive, and Prompt-Based Methods for Job Title and Skill Matching](https://mlsanity.com/paper/2506.19058/) (2025)
- [UniSkill: A Dataset for Matching University Curricula to Professional Competencies](https://mlsanity.com/paper/2603.03134/) (2026)
- [Scalable Identification and Prioritization of Requisition-Specific Personal Competencies Using Large Language Models](https://mlsanity.com/paper/2604.00006/) (2026)

## Abstract (arXiv)

The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill classification, job title classification, and de-identification. While some approaches have been developed that are specific to the job market domain, there is a lack of generalized, multilingual models and benchmarks for these tasks. In this study, we introduce a language model called ESCOXLM-R, based on XLM-R, which uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy, covering 27 languages. The pre-training objectives for ESCOXLM-R include dynamic masked language modeling and a novel additional objective for inducing multilingual taxonomical ESCO relations. We comprehensively evaluate the performance of ESCOXLM-R on 6 sequence labeling and 3 classification tasks in 4 languages and find that it achieves state-of-the-art results on 6 out of 9 datasets. Our analysis reveals that ESCOXLM-R performs better on short spans and outperforms XLM-R on entity-level and surface-level span-F1, likely due to ESCO containing short skill and occupation titles, and encoding information on the entity-level.