# CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting

arXiv: https://arxiv.org/abs/2104.03642 · HTML: https://mlsanity.com/paper/2104.03642/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Multimodal Machine Learning-based Knee Osteoarthritis Progression Prediction from Plain Radiographs and Clinical Data](https://mlsanity.com/paper/1904.06236/) (2019)
- [Semixup: In- and Out-of-Manifold Regularization for Deep Semi-Supervised Knee Osteoarthritis Severity Grading from Plain Radiographs](https://mlsanity.com/paper/2003.01944/) (2020)
- [Multi-classifier prediction of knee osteoarthritis progression from incomplete imbalanced longitudinal data](https://mlsanity.com/paper/1909.13408/) (2019)

## Built on by (most cited first, 4 of 4)

- [Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Data](https://mlsanity.com/paper/2210.13889/) (2022)
- [SCouT: Synthetic Counterfactuals via Spatiotemporal Transformers for Actionable Healthcare](https://mlsanity.com/paper/2207.04208/) (2022)
- [End-To-End Prediction of Knee Osteoarthritis Progression With Multi-Modal Transformers](https://mlsanity.com/paper/2307.00873/) (2023)
- [Toward Cost-efficient Adaptive Clinical Trials in Knee Osteoarthritis with Reinforcement Learning](https://mlsanity.com/paper/2408.02349/) (2024)

## Abstract (arXiv)

In medical applications, deep learning methods are built to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decision-making process with two agents -- a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances. Our code is publicly available at urlhttps://github.com/MIPT-Oulu/CLIMAT.