# Covariance Descriptors Meet General Vision Encoders: Riemannian Deep Learning for Medical Image Classification

arXiv: https://arxiv.org/abs/2511.04190 · HTML: https://mlsanity.com/paper/2511.04190/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 6 Nov 2025.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [A Riemannian Network for SPD Matrix Learning](https://mlsanity.com/paper/1608.04233/) (2016)
- [MedViT: A Robust Vision Transformer for Generalized Medical Image Classification](https://mlsanity.com/paper/2302.09462/) (2023)
- [geomstats: a Python Package for Riemannian Geometry in Machine Learning](https://mlsanity.com/paper/1805.08308/) (2018)
- [Spatio-Temporal Covariance Descriptors for Action and Gesture Recognition](https://mlsanity.com/paper/1303.6021/) (2013)
- [Ricci flow-based brain surface covariance descriptors for diagnosing Alzheimer's disease](https://mlsanity.com/paper/2403.06645/) (2024)
- [High Accuracy Classification of White Blood Cells using TSLDA Classifier and Covariance Features](https://mlsanity.com/paper/1906.05131/) (2019)

## Built on by (most cited first, 1 of 1)

- [Cov2Pose: Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation](https://mlsanity.com/paper/2603.19961/) (2026)

## Abstract (arXiv)

Covariance descriptors capture second-order statistics of image features. They have shown strong performance in general computer vision tasks, but remain underexplored in medical imaging. We investigate their effectiveness for both conventional and learning-based medical image classification, with a particular focus on SPDNet, a classification network specifically designed for symmetric positive definite (SPD) matrices. We propose constructing covariance descriptors from features extracted by pre-trained general vision encoders (GVEs) and comparing them with handcrafted descriptors. Two GVEs - DINOv2 and MedSAM - are evaluated across eleven binary and multi-class datasets from the MedMNSIT benchmark. Our results show that covariance descriptors derived from GVE features consistently outperform those derived from handcrafted features. Moreover, SPDNet yields superior performance to state-of-the-art methods when combined with DINOv2 features. Our findings highlight the potential of combining covariance descriptors with powerful pretrained vision encoders for medical image analysis.