# CNN Models for Microphone Array Covariance Matrix Upsampling and Acoustic Imaging

arXiv: https://arxiv.org/abs/2607.01295 · HTML: https://mlsanity.com/paper/2607.01295/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 27 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Builds on (most cited first, 6 of 6)

- [Deep Back-Projection Networks For Super-Resolution](https://mlsanity.com/paper/1803.02735/) (2018)
- [STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Events](https://mlsanity.com/paper/2306.09126/) (2023)
- [Robust DOA estimation using deep acoustic imaging](https://mlsanity.com/paper/2401.08717/) (2024)
- [Neural Ambisonics encoding for compact irregular microphone arrays](https://mlsanity.com/paper/2401.05916/) (2024)
- [A Physics-Informed Neural Network-Based Approach for the Spatial Upsampling of Spherical Microphone Arrays](https://mlsanity.com/paper/2407.18732/) (2024)
- [Latent Acoustic Mapping for Direction of Arrival Estimation: A Self-Supervised Approach](https://mlsanity.com/paper/2507.07066/) (2025)

## Abstract (arXiv)

Acoustic imaging visualization is a core methodology in acoustics, enabling spatial analysis of sound sources and acoustic scenes. However, limited sensor availability in practical systems motivate approaches that enhance spatial resolution without increasing the hardware complexity. In this paper, we focus on upsampling virtually a tetrahedral 4-microphone array to a spherical 32-microphone array by estimating the covariance matrices of the channels employing deep learning techniques. Five neural network architectures are investigated for covariance upsampling for acoustic imaging using the real-world STARSS23 dataset. These models are developed to estimate a 32-microphone, time-frequency covariance matrix from a 4-microphone input covariance representation. The proposed architectures are based on 2D convolutional layers to capture the underlying spatial-spectral structure of covariance matrices, and are further enhanced with frequency dynamic convolution to model their frequency-dependent properties. The proposed architectures are evaluated in terms of root mean square error (RMSE) and using delay-and-sum beamforming acoustic imaging. Quantitative results show that all models outperform a random-guess baseline, which yields an RMSE of 0.548, with the best-performing architecture achieving an RMSE of 0.432. We analyze qualitatively the performance of the proposed models through beamforming heatmap visualizations derived from the 4-channel input covariance, the 32-channel ground truth, and the predicted 32-channel covariance matrices. These results demonstrate that covariance upsampling significantly enhances the effective performance of the 4-channel microphone array, producing sound maps that closely resemble those obtained with the 32-channel array.