# When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?

arXiv: https://arxiv.org/abs/2408.11854 · HTML: https://mlsanity.com/paper/2408.11854/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 48 references that are in the map.
- First posted on arXiv 15 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Demystifying Prompts in Language Models via Perplexity Estimation](https://mlsanity.com/paper/2212.04037/) (2022)
- [MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III](https://mlsanity.com/paper/1907.08322/) (2019)
- [CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks](https://mlsanity.com/paper/2111.08585/) (2021)
- [Integrated multimodal artificial intelligence framework for healthcare applications](https://mlsanity.com/paper/2202.12998/) (2022)
- [Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review](https://mlsanity.com/paper/2010.02809/) (2020)
- [Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data](https://mlsanity.com/paper/2311.02216/) (2023)
- [What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary](https://mlsanity.com/paper/2212.10380/) (2022)
- [Large Language Models are Complex Table Parsers](https://mlsanity.com/paper/2312.11521/) (2023)
- [What Makes Pre-trained Language Models Better Zero-shot Learners?](https://mlsanity.com/paper/2209.15206/) (2022)
- [Unified Language Representation for Question Answering over Text, Tables, and Images](https://mlsanity.com/paper/2306.16762/) (2023)
- [Estimating Large Language Model Capabilities without Labeled Test Data](https://mlsanity.com/paper/2305.14802/) (2023)
- [DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely](https://mlsanity.com/paper/2212.10013/) (2022)

## Built on by (most cited first, 12 of 12)

- [Large Language Models are Powerful Electronic Health Record Encoders](https://mlsanity.com/paper/2502.17403/) (2025)
- [FoNE: Precise Single-Token Number Embeddings via Fourier Features](https://mlsanity.com/paper/2502.09741/) (2025)
- [Large Language Models with Temporal Reasoning for Longitudinal Clinical Summarization and Prediction](https://mlsanity.com/paper/2501.18724/) (2025)
- [ChronoFormer: Time-Aware Transformer Architectures for Structured Clinical Event Modeling](https://mlsanity.com/paper/2504.07373/) (2025)
- [Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting](https://mlsanity.com/paper/2503.04013/) (2025)
- [MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling](https://mlsanity.com/paper/2508.05492/) (2025)
- [LLMs as World Models: Data-Driven and Human-Centered Pre-Event Simulation for Disaster Impact Assessment](https://mlsanity.com/paper/2506.06355/) (2025)
- [Can we generate portable representations for clinical time series data using LLMs?](https://mlsanity.com/paper/2603.23987/) (2026)
- [Position Paper On Diagnostic Uncertainty Estimation from Large Language Models: Next-Word Probability Is Not Pre-test Probability](https://mlsanity.com/paper/2411.04962/) (2024)
- [Integrating Dynamical Systems Learning with Foundational Models: A Meta-Evolutionary AI Framework for Clinical Trials](https://mlsanity.com/paper/2506.14782/) (2025)
- [Benchmarking LLMs for Predictive Applications in the Intensive Care Units](https://mlsanity.com/paper/2512.20520/) (2025)
- [Learning to Select Source-Traceable Evidence for Language-Model Prediction from Irregular Clinical Time Series](https://mlsanity.com/paper/2605.20292/) (2026)

## Abstract (arXiv)

The introduction of Large Language Models (LLMs) has advanced data representation and analysis, bringing significant progress in their use for medical questions and answering. Despite these advancements, integrating tabular data, especially numerical data pivotal in clinical contexts, into LLM paradigms has not been thoroughly explored. In this study, we examine the effectiveness of vector representations from last hidden states of LLMs for medical diagnostics and prognostics using electronic health record (EHR) data. We compare the performance of these embeddings with that of raw numerical EHR data when used as feature inputs to traditional machine learning (ML) algorithms that excel at tabular data learning, such as eXtreme Gradient Boosting. We focus on instruction-tuned LLMs in a zero-shot setting to represent abnormal physiological data and evaluating their utilities as feature extractors to enhance ML classifiers for predicting diagnoses, length of stay, and mortality. Furthermore, we examine prompt engineering techniques on zero-shot and few-shot LLM embeddings to measure their impact comprehensively. Although findings suggest the raw data features still prevails in medical ML tasks, zero-shot LLM embeddings demonstrate competitive results, suggesting a promising avenue for future research in medical applications.