# Cross-Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptation

arXiv: https://arxiv.org/abs/2511.05841 · HTML: https://mlsanity.com/paper/2511.05841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 53 references that are in the map.
- First posted on arXiv 8 Nov 2025.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)
- [WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation](https://mlsanity.com/paper/2303.14814/) (2023)
- [AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection](https://mlsanity.com/paper/2310.18961/) (2023)
- [Task Residual for Tuning Vision-Language Models](https://mlsanity.com/paper/2211.10277/) (2022)
- [Diffusion Model as Representation Learner](https://mlsanity.com/paper/2308.10916/) (2023)
- [Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts](https://mlsanity.com/paper/2403.06495/) (2024)
- [BMAD: Benchmarks for Medical Anomaly Detection](https://mlsanity.com/paper/2306.11876/) (2023)
- [Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images](https://mlsanity.com/paper/2403.12570/) (2024)
- [Encoding Structure-Texture Relation with P-Net for Anomaly Detection in Retinal Images](https://mlsanity.com/paper/2008.03632/) (2020)
- [BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models](https://mlsanity.com/paper/2411.15232/) (2024)
- [Proxy-bridged Image Reconstruction Network for Anomaly Detection in Medical Images](https://mlsanity.com/paper/2110.01761/) (2021)
- [Aligning Human Knowledge with Visual Concepts Towards Explainable Medical Image Classification](https://mlsanity.com/paper/2406.05596/) (2024)
- [Multi-scale Cross-restoration Framework for Electrocardiogram Anomaly Detection](https://mlsanity.com/paper/2308.01639/) (2023)
- [Sequence-based Dynamic Handwriting Analysis for Parkinson's Disease Detection with One-dimensional Convolutions and BiGRUs](https://mlsanity.com/paper/2101.09461/) (2021)
- [How word semantics and phonology affect handwriting of Alzheimer's patients: a machine learning based analysis](https://mlsanity.com/paper/2307.04762/) (2023)
- [Hybrid Transformer for Early Alzheimer's Detection: Integration of Handwriting-Based 2D Images and 1D Signal Features](https://mlsanity.com/paper/2410.10547/) (2024)

## Abstract (arXiv)

Alzheimer's disease (AD) is a prevalent neurodegenerative disorder for which early detection is critical. Handwriting, which can be disrupted by subtle motor and cognitive decline, provides a non-invasive and cost-effective window for AD screening. Existing handwriting-based AD studies mostly rely on online trajectories and hand-crafted features, while the influence of handwriting task type on diagnostic performance and cross-task generalization remains underexplored. Meanwhile, large-scale vision--language models have demonstrated strong transfer and adaptation ability in natural-image anomaly detection and several medical modalities, such as chest X-ray and brain MRI. However, handwriting-based disease detection remains unexplored within this paradigm. To address this gap, we introduce a lightweight Cross-Layer Fusion Adapter (CLFA) framework that repurposes Contrastive Language--Image Pre-training (CLIP) for handwriting-based AD screening. CLFA inserts multi-level adapters into a frozen visual encoder, combining cross-layer feature fusion with depthwise 2D convolution on patch grids to capture both local stroke irregularities and higher-level handwriting structure. This design progressively aligns pretrained vision--language representations with AD-related handwriting cues and supports transfer from supervised source tasks to task-disjoint unseen target tasks. On the Darwin dataset, under the subject-disjoint cross-task protocol, averaged over all 600 task-disjoint source-target pairs, CLFA achieves 74.63% AUC, 74.85% accuracy, and 73.72% F1 score, outperforming the best competing model by 2.15, 1.79, and 1.87 percentage points, respectively.