# Labrador: Exploring the Limits of Masked Language Modeling for Laboratory Data

arXiv: https://arxiv.org/abs/2312.11502 · HTML: https://mlsanity.com/paper/2312.11502/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 9 Dec 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [xVal: A Continuous Numerical Tokenization for Scientific Language Models](https://mlsanity.com/paper/2310.02989/) (2023)
- [Evaluating Progress on Machine Learning for Longitudinal Electronic Healthcare Data](https://mlsanity.com/paper/2010.01149/) (2020)
- [Evaluation of Embeddings of Laboratory Test Codes for Patients at a Cancer Center](https://mlsanity.com/paper/1907.09600/) (2019)

## Built on by (most cited first, 9 of 9)

- [MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models](https://mlsanity.com/paper/2407.12309/) (2024)
- [Zero-shot Medical Event Prediction Using a Generative Pre-trained Transformer on Electronic Health Records](https://mlsanity.com/paper/2503.05893/) (2025)
- [Representation Learning of Lab Values via Masked AutoEncoders](https://mlsanity.com/paper/2501.02648/) (2025)
- [SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems](https://mlsanity.com/paper/2405.19653/) (2024)
- [CURENet: Combining Unified Representations for Efficient Chronic Disease Prediction](https://mlsanity.com/paper/2511.11423/) (2025)
- [Coefficient of Variation Masking: A Volatility-Aware Strategy for EHR Foundation Models](https://mlsanity.com/paper/2512.05216/) (2025)
- [Temporal Fusion Nexus: A task-agnostic multi-modal embedding model for clinical narratives and irregular time series in post-kidney transplant care](https://mlsanity.com/paper/2601.08503/) (2026)
- [Learning Representations from Incomplete EHR Data with Dual-Masked Autoencoding](https://mlsanity.com/paper/2602.15159/) (2026)
- [How Should Transformers Encode Numeric Values in Electronic Health Records?](https://mlsanity.com/paper/2607.01391/) (2026)

## Abstract (arXiv)

In this work we introduce Labrador, a pre-trained Transformer model for laboratory data. Labrador and BERT were pre-trained on a corpus of 100 million lab test results from electronic health records (EHRs) and evaluated on various downstream outcome prediction tasks. Both models demonstrate mastery of the pre-training task but neither consistently outperform XGBoost on downstream supervised tasks. Our ablation studies reveal that transfer learning shows limited effectiveness for BERT and achieves marginal success with Labrador. We explore the reasons for the failure of transfer learning and suggest that the data generating process underlying each patient cannot be characterized sufficiently using labs alone, among other factors. We encourage future work to focus on joint modeling of multiple EHR data categories and to include tree-based baselines in their evaluations.