# CASHformer: Cognition Aware SHape Transformer for Longitudinal Analysis

arXiv: https://arxiv.org/abs/2207.02091 · HTML: https://mlsanity.com/paper/2207.02091/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [Pretrained Transformers as Universal Computation Engines](https://mlsanity.com/paper/2103.05247/) (2021)
- [SpiralNet++: A Fast and Highly Efficient Mesh Convolution Operator](https://mlsanity.com/paper/1911.05856/) (2019)
- [Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers](https://mlsanity.com/paper/2002.11794/) (2020)
- [Longitudinal Self-Supervised Learning](https://mlsanity.com/paper/2006.06930/) (2020)
- [Few-Shot Domain Adaptation with Polymorphic Transformers](https://mlsanity.com/paper/2107.04805/) (2021)
- [TransforMesh: A Transformer Network for Longitudinal modeling of Anatomical Meshes](https://mlsanity.com/paper/2109.00532/) (2021)
- [Geometric Deep Learning on Anatomical Meshes for the Prediction of Alzheimer's Disease](https://mlsanity.com/paper/2104.10047/) (2021)

## Built on by (most cited first, 1 of 1)

- [TractGraphFormer: Anatomically Informed Hybrid Graph CNN-Transformer Network for Interpretable Sex and Age Prediction from Diffusion MRI Tractography](https://mlsanity.com/paper/2407.08883/) (2024)

## Abstract (arXiv)

Modeling temporal changes in subcortical structures is crucial for a better understanding of the progression of Alzheimer's disease (AD). Given their flexibility to adapt to heterogeneous sequence lengths, mesh-based transformer architectures have been proposed in the past for predicting hippocampus deformations across time. However, one of the main limitations of transformers is the large amount of trainable parameters, which makes the application on small datasets very challenging. In addition, current methods do not include relevant non-image information that can help to identify AD-related patterns in the progression. To this end, we introduce CASHformer, a transformer-based framework to model longitudinal shape trajectories in AD. CASHformer incorporates the idea of pre-trained transformers as universal compute engines that generalize across a wide range of tasks by freezing most layers during fine-tuning. This reduces the number of parameters by over 90% with respect to the original model and therefore enables the application of large models on small datasets without overfitting. In addition, CASHformer models cognitive decline to reveal AD atrophy patterns in the temporal sequence. Our results show that CASHformer reduces the reconstruction error by 73% compared to previously proposed methods. Moreover, the accuracy of detecting patients progressing to AD increases by 3% with imputing missing longitudinal shape data.