mlsanityData to 14 Oct 2026

Adapting Pretrained Language Models for Solving Tabular Prediction Problems in the Electronic Health Record

arXiv 2303.14920 ↗First posted 27 Mar 2023cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We propose an approach for adapting the DeBERTa model for electronic health record (EHR) tasks using domain adaptation. We pretrain a small DeBERTa model on a dataset consisting of MIMIC-III discharge summaries, clinical notes, radiology reports, and PubMed abstracts. We compare this model's performance with a DeBERTa model pre-trained on clinical texts from our institutional EHR (MeDeBERTa) and an XGBoost model. We evaluate performance on three benchmark tasks for emergency department outcomes using the MIMIC-IV-ED dataset. We preprocess the data to convert it into text format and generate four versions of the original datasets to compare data processing and data inclusion. The results show that our proposed approach outperforms the alternative models on two of three tasks (p<0.001) and matches performance on the third task, with the use of descriptive columns improving performance over the original column names.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

19 of 28 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  10. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  11. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  12. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
7 more
  1. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  2. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 159
  3. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  4. 2018Did the Model Understand the Question?cited by 60
  5. 2022Transfer Learning with Deep Tabular Modelscited by 52
  6. 2021Benchmarking emergency department triage prediction models with machine learning and large public electronic health recordscited by 31
  7. 2022PTab: Using the Pre-trained Language Model for Modeling Tabular Datacited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 0; 2026: 1.

5 papers.

  1. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  2. 2024Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Modelscited by 36
  3. 2024Elephants Never Forget: Testing Language Models for Memorization of Tabular Datacited by 19
  4. 2024Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolutioncited by 19
  5. 2026SL-S4Wave: Self-Supervised Learning of Physiological Waveforms with Structured State Space Modelscited by 0