# Self-Supervised Learning for Covariance Estimation

arXiv: https://arxiv.org/abs/2403.08662 · HTML: https://mlsanity.com/paper/2403.08662/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 30 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SpectralGPT: Spectral Remote Sensing Foundation Model](https://mlsanity.com/paper/2311.07113/) (2023)
- [Model-Based Deep Learning](https://mlsanity.com/paper/2012.08405/) (2020)
- [Robust Estimation of Structured Covariance Matrix for Heavy-Tailed Elliptical Distributions](https://mlsanity.com/paper/1506.05215/) (2015)
- [Covariance Prediction via Convex Optimization](https://mlsanity.com/paper/2101.12416/) (2021)
- [New Methods for MLE of Toeplitz Structured Covariance Matrices with Applications to RADAR Problems](https://mlsanity.com/paper/2307.03923/) (2023)
- [Unsupervised Learning Discriminative MIG Detectors in Nonhomogeneous Clutter](https://mlsanity.com/paper/2204.11278/) (2022)

## Abstract (arXiv)

We consider the use of deep learning for covariance estimation. We propose to globally learn a neural network that will then be applied locally at inference time. Leveraging recent advancements in self-supervised foundational models, we train the network without any labeling by simply masking different samples and learning to predict their covariance given their surrounding neighbors. The architecture is based on the popular attention mechanism. Its main advantage over classical methods is the automatic exploitation of global characteristics without any distributional assumptions or regularization. It can be pre-trained as a foundation model and then be repurposed for various downstream tasks, e.g., adaptive target detection in radar or hyperspectral imagery.