# Count-Based Approaches Remain Strong: A Benchmark Against Transformer and LLM Pipelines on Structured EHR

arXiv: https://arxiv.org/abs/2511.00782 · HTML: https://mlsanity.com/paper/2511.00782/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 2 Nov 2025.

## Builds on (most cited first, 16 of 16)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://mlsanity.com/paper/2307.02028/) (2023)
- [Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data](https://mlsanity.com/paper/2001.05295/) (2020)
- [MOTOR: A Time-To-Event Foundation Model For Structured Medical Records](https://mlsanity.com/paper/2301.03150/) (2023)
- [Multi-domain Clinical Natural Language Processing with MedCAT: the Medical Concept Annotation Toolkit](https://mlsanity.com/paper/2010.01165/) (2020)
- [Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events](https://mlsanity.com/paper/2306.11547/) (2023)
- [Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs](https://mlsanity.com/paper/2412.16178/) (2024)
- [ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?](https://mlsanity.com/paper/2411.06469/) (2024)
- [A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records](https://mlsanity.com/paper/2311.11483/) (2023)
- [MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling](https://mlsanity.com/paper/2508.05492/) (2025)
- [MedRep: Medical Concept Representation for General Electronic Health Record Foundation Models](https://mlsanity.com/paper/2504.08329/) (2025)
- [Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs](https://mlsanity.com/paper/2508.14817/) (2025)

## Built on by (most cited first, 3 of 3)

- [TabPFN-3: Technical Report](https://mlsanity.com/paper/2605.13986/) (2026)
- [Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraints](https://mlsanity.com/paper/2604.01841/) (2026)
- [Structured Evidence Routing for Incident Risk Prediction from Multimodal Longitudinal EHRs](https://mlsanity.com/paper/2608.26191/) (2026)

## Abstract (arXiv)

Structured electronic health records (EHR) are essential for clinical prediction. While count-based learners continue to perform strongly on such data, no benchmarking has directly compared them against more recent mixture-of-agents LLM pipelines, which have been reported to outperform single LLMs in various NLP tasks. In this study, we evaluated three categories of methodologies for EHR prediction using the EHRSHOT dataset: count-based models built from ontology roll-ups with two time bins, based on LightGBM and the tabular foundation model TabPFN; a pretrained sequential transformer (CLMBR); and a mixture-of-agents pipeline that converts tabular histories to natural-language summaries followed by a text classifier. We assessed eight outcomes using the EHRSHOT dataset. Across the eight evaluation tasks, head-to-head wins were largely split between the count-based and the mixture-of-agents methods. Given their simplicity and interpretability, count-based models remain a strong candidate for structured EHR benchmarking. The source code is available at: https://github.com/cristea-lab/Structured_EHR_Benchmark.