KenCoh: A Ranked-Based Canonical Coherence

This work is inspired by the problem of characterizing a dependence measure between two cortical regions of the brain where each region contains multiple signal recordings from several neurons or channels (e.g., inhibitory and excitatory neurons). The goal is to identify differences in the structure of brain functional connectivity between known brain states. An exploratory tool for studying the dependence between two random vectors is via canonical correlation analysis. However, these are limited to only capturing linear associations and are sensitive to outlier observations. Mitigating these limitations is crucial because brain functional connectivity is likely to be more complex than linear, and brain signals may exhibit heavy-tailed properties. To overcome these limitations, we develop a robust method, Kendall's tau-based canonical coherence (KenCoh), to learn connectivity structure among neuronal signals filtered at given frequency bands. Our simulation study demonstrates that KenCoh is competitive with the moment-based estimator and outperforms the latter when the underlying distributions are heavy-tailed. We apply our method to EEG recordings from a virtual-reality driving experiment and to calcium imaging recordings in inhibitory and excitatory neurons of the auditory cortex in mice subjected to sound stimuli. Our findings reveal distinct regional dependencies across frequency bands and brain states.