# Multi-objective Representation for Numbers in Clinical Narratives: A CamemBERT-Bio-Based Alternative to Large-Scale LLMs

arXiv: https://arxiv.org/abs/2405.18448 · HTML: https://mlsanity.com/paper/2405.18448/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 21 of 21)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Do Language Embeddings Capture Scales?](https://mlsanity.com/paper/2010.05345/) (2020)
- [Linear algebra with transformers](https://mlsanity.com/paper/2112.01898/) (2021)
- [FiNER: Financial Numeric Entity Recognition for XBRL Tagging](https://mlsanity.com/paper/2203.06482/) (2022)
- [A Comparison of LSTM and BERT for Small Corpus](https://mlsanity.com/paper/2009.05451/) (2020)
- [Comparative Analysis of Text Classification Approaches in Electronic Health Records](https://mlsanity.com/paper/2005.06624/) (2020)
- [Early Prediction of Acute Kidney Injury in Critical Care Setting Using Clinical Notes](https://mlsanity.com/paper/1811.02757/) (2018)
- [Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL](https://mlsanity.com/paper/2405.06410/) (2024)
- [Detecting of a Patient's Condition From Clinical Narratives Using Natural Language Representation](https://mlsanity.com/paper/2104.03969/) (2021)
- [Adaptation of Autoencoder for Sparsity Reduction From Clinical Notes Representation Learning](https://mlsanity.com/paper/2209.12831/) (2022)
- [Improving Transformer Performance for French Clinical Notes Classification Using Mixture of Experts on a Limited Dataset](https://mlsanity.com/paper/2303.12892/) (2023)
- [EmbNum: Semantic labeling for numerical values with deep metric learning](https://mlsanity.com/paper/1807.01367/) (2018)
- [DistilCamemBERT: a distillation of the French model CamemBERT](https://mlsanity.com/paper/2205.11111/) (2022)
- [Extracting Physiological Numeric Values from French Pediatric ICU Notes: A Multi-Objective Representation Learning Approach](https://mlsanity.com/paper/2404.10171/) (2024)

## Built on by (most cited first, 1 of 1)

- [Tokenized Key-Gated Adapter Routing: A Secure Access Control Mechanism Against Private Data Leakage in LLMs](https://mlsanity.com/paper/2610.00309/) (2026)

## Abstract (arXiv)

The processing of numerical values is a rapidly developing area in the field of Language Models (LLMs). Despite numerous advancements achieved by previous research, significant challenges persist, particularly within the healthcare domain. This paper investigates the limitations of Transformer models in understanding numerical values. textitObjective: this research aims to categorize numerical values extracted from medical documents into eight specific physiological categories using CamemBERT-bio. textitMethods: In a context where scalable methods and Large Language Models (LLMs) are emphasized, we explore lifting the limitations of transformer-based models. We examine two strategies: fine-tuning CamemBERT-bio on a small medical dataset, integrating Label Embedding for Self-Attention (LESA), and combining LESA with additional enhancement techniques such as Xval. Given that CamemBERT-bio is already pre-trained on a large medical dataset, the first approach aims to update its encoder with the newly added label embeddings technique. In contrast, the second approach seeks to develop multiple representations of numbers (contextual and magnitude-based) to achieve more robust number embeddings. textitResults: As anticipated, fine-tuning the standard CamemBERT-bio on our small medical dataset did not improve F1 scores. However, significant improvements were observed with CamemBERT-bio + LESA, resulting in an over 13% increase. Similar enhancements were noted when combining LESA with Xval, outperforming conventional methods and giving comparable results to GPT-4 textitConclusions and Novelty: This study introduces two innovative techniques for handling numerical data, which are also applicable to other modalities. We illustrate how these techniques can improve the performance of Transformer-based models, achieving more reliable classification results even with small datasets.