# MedTVL: Harnessing Vision and Language for Medical Time Series Classification

arXiv: https://arxiv.org/abs/2608.28605 · HTML: https://mlsanity.com/paper/2608.28605/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 7 Jul 2026.

## Builds on (most cited first, 17 of 17)

- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [UniTS: A Unified Multi-Task Time Series Model](https://mlsanity.com/paper/2403.00131/) (2024)
- [MultiRocket: Multiple pooling operators and transformations for fast and effective time series classification](https://mlsanity.com/paper/2102.00457/) (2021)
- [Time Series as Images: Vision Transformer for Irregularly Sampled Time Series](https://mlsanity.com/paper/2303.12799/) (2023)
- [Frozen Language Model Helps ECG Zero-Shot Learning](https://mlsanity.com/paper/2303.12311/) (2023)
- [Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classification](https://mlsanity.com/paper/2405.19363/) (2024)
- [Time Series Contrastive Learning with Information-Aware Augmentations](https://mlsanity.com/paper/2303.11911/) (2023)
- [TimesURL: Self-supervised Contrastive Learning for Universal Time Series Representation Learning](https://mlsanity.com/paper/2312.15709/) (2023)
- [MedualTime: A Dual-Adapter Language Model for Medical Time Series-Text Multimodal Learning](https://mlsanity.com/paper/2406.06620/) (2024)
- [MedGNN: Towards Multi-resolution Spatiotemporal Graph Learning for Medical Time Series Classification](https://mlsanity.com/paper/2502.04515/) (2025)
- [MedSpaformer: a Transferable Transformer with Multi-granularity Token Sparsification for Medical Time Series Classification](https://mlsanity.com/paper/2503.15578/) (2025)

## Abstract (arXiv)

Recent advancements in multimodal learning for medical time series (MedTS) classification highlight the benefits of integrating complementary modalities for clinical decision. However, existing methods typically focus on bi-modal interactions (e.g., time series and text), leaving the tri-modal synergy between time series, vision, and language largely unexplored. Inspired by diagnostic practice synergizing numerical assessment, visual inspection and clinical context, we introduce MedTVL, a text-guided dual-pathway architecture tailored for MedTS classification. Specifically, it synergizes a convolution-based temporal pathway for fine-grained temporal dynamics from raw numerical sequences and a transformer-based visual pathway for holistic morphological structures from time-series-derived images. Such combination of cross-modal and architectural heterogeneity provides a comprehensive diagnostic perspective. To further resolve potential diagnostic ambiguity, both pathways are guided by adaptive medical textual semantics. Finally, a Mixture-of-Experts mechanism dynamically routes each instance to specialized fusion experts, capturing instance-specific reliance on the temporal and visual pathway outputs. In addition, MedTVL supports multimodal contrastive learning to mitigate the clinical label scarcity challenge. Extensive experiments across multiple medical datasets and tasks, spanning supervised, few-shot, and contrastive learning settings, demonstrate the superiority and transferability of MedTVL, highlighting its potential for robust clinical decision support.