# ATHENA: Knowledge-guided agentic neural architecture search for AutoFormer-based electronic health record modeling

arXiv: https://arxiv.org/abs/2608.21712 · HTML: https://mlsanity.com/paper/2608.21712/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 22 Aug 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [AutoFormer: Searching Transformers for Visual Recognition](https://mlsanity.com/paper/2107.00651/) (2021)
- [The Evolved Transformer](https://mlsanity.com/paper/1901.11117/) (2019)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://mlsanity.com/paper/2307.02028/) (2023)
- [LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization](https://mlsanity.com/paper/2306.01102/) (2023)
- [CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks](https://mlsanity.com/paper/2111.08585/) (2021)
- [Can GPT-4 Perform Neural Architecture Search?](https://mlsanity.com/paper/2304.10970/) (2023)
- [Multi-layer Representation Learning for Medical Concepts](https://mlsanity.com/paper/1602.05568/) (2016)
- [NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search](https://mlsanity.com/paper/2105.14444/) (2021)
- [Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records](https://mlsanity.com/paper/2106.11360/) (2021)
- [DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency](https://mlsanity.com/paper/2403.06197/) (2024)
- [MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records](https://mlsanity.com/paper/2102.02340/) (2021)
- [AutoBERT-Zero: Evolving BERT Backbone from Scratch](https://mlsanity.com/paper/2107.07445/) (2021)
- [Recent Advances in Predictive Modeling with Electronic Health Records](https://mlsanity.com/paper/2402.01077/) (2024)
- [A Comprehensive Survey of Electronic Health Record Modeling: From Deep Learning Approaches to Large Language Models](https://mlsanity.com/paper/2507.12774/) (2025)
- [CoLLM-NAS: Collaborative Large Language Models for Efficient Knowledge-Guided Neural Architecture Search](https://mlsanity.com/paper/2509.26037/) (2025)
- [MetaLLMix : An XAI Aided LLM-Meta-learning Based Approach for Hyper-parameters Optimization](https://mlsanity.com/paper/2509.09387/) (2025)
- [DT-BEHRT: Disease Trajectory-aware Transformer for Interpretable Patient Representation Learning](https://mlsanity.com/paper/2603.10180/) (2026)

## Abstract (arXiv)

Transformer architectures for clinical prediction from electronic health records (EHRs) often require costly manual tuning and may vary across hospitals and tasks. We introduce ATHENA (Agentic Transfer across Hospitals for EHR Neural Architecture Search), a knowledge-guided agentic neural architecture search framework for Transformer-based EHR modeling. A weight-sharing supernet is pretrained once per hospital, allowing candidate architectures to inherit weights and be evaluated through fine-tuning rather than independent pretraining. A two-layer cross-hospital prior supplies high-performing source architecture examples and task-specific component preferences summarized by SHapley Additive exPlanations (SHAP)-based meta-regression. These priors guide a multi-agent large language model search using target-hospital validation feedback. Across six clinical prediction tasks at one held-out OneFlorida+ site and one external MIMIC-IV site, ATHENA significantly outperforms all four baselines in 9 of 12 site-task evaluations under a strict equal-compute comparison. In a separate comparison using a common pretrained AutoFormer supernet to isolate search strategy performance, ATHENA ranks first in 9 of 12 evaluations at a maximum search budget of 30 candidate evaluations. It also shows more consistent architecture selection across repeated searches. ATHENA supports more efficient architecture selection for Transformer-based clinical prediction models.