# An Unsupervised Natural Language Processing Pipeline for Assessing Referral Appropriateness

arXiv: https://arxiv.org/abs/2501.14701 · HTML: https://mlsanity.com/paper/2501.14701/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 54 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Enhancing Clinical Concept Extraction with Contextual Embeddings](https://mlsanity.com/paper/1902.08691/) (2019)
- [Influence of various text embeddings on clustering performance in NLP](https://mlsanity.com/paper/2305.03144/) (2023)
- [An Analysis of Word2Vec for the Italian Language](https://mlsanity.com/paper/2001.09332/) (2020)
- [Understanding patient complaint characteristics using contextual clinical BERT embeddings](https://mlsanity.com/paper/2002.05902/) (2020)
- [Automatic identification of diagnosis from hospital discharge letters via weakly supervised Natural Language Processing](https://mlsanity.com/paper/2410.15051/) (2024)

## Abstract (arXiv)

Objective: Assessing the appropriateness of diagnostic referrals is critical for improving healthcare efficiency and reducing unnecessary procedures. However, this task becomes challenging when referral reasons are recorded only as free text rather than structured codes, like in the Italian NHS. To address this gap, we propose a fully unsupervised Natural Language Processing (NLP) pipeline capable of extracting and evaluating referral reasons without relying on labelled datasets. Methods: Our pipeline leverages Transformer-based embeddings pre-trained on Italian medical texts to cluster referral reasons and assess their alignment with appropriateness guidelines. It operates in an unsupervised setting and is designed to generalize across different examination types. We analyzed two complete regional datasets from the Lombardy Region (Italy), covering all referrals between 2019 and 2021 for venous echocolordoppler of the lower limbs (ECD;n=496,971; development) and flexible endoscope colonoscopy (FEC; n=407,949; testing only). For both, a random sample of 1,000 referrals was manually annotated to measure performance. Results: The pipeline achieved high performance in identifying referral reasons (Prec=92.43% (ECD), 93.59% (FEC); Rec=83.28% (ECD), 92.70% (FEC)) and appropriateness (Prec=93.58% (ECD), 94.66% (FEC); Rec=91.52% (ECD), 93.96% (FEC)). At the regional level, the analysis identified relevant inappropriate referral groups and variation across contexts, findings that informed a new Lombardy Region resolution to reinforce guideline adherence. Conclusions: This study presents a robust, scalable, unsupervised NLP pipeline for assessing referral appropriateness in large, real-world datasets. It demonstrates how such data can be effectively leveraged, providing public health authorities with a deployable AI tool to monitor practices and support evidence-based policy.