Off-Grid DOA Estimation Using Sparse Bayesian Learning in MIMO Radar With Unknown Mutual Coupling

In the practical radar with multiple antennas, the antenna imperfections degrade the system performance. In this paper, the problem of estimating the direction of arrival (DOA) in multiple-input and multiple-output (MIMO) radar system with unknown mutual coupling effect between antennas is investigated. To exploit the target sparsity in the spatial domain, the compressed sensing (CS)-based methods have been proposed by discretizing the detection area and formulating the dictionary matrix, so an off-grid gap is caused by the discretization processes. In this paper, different from the present DOA estimation methods, both the off-grid gap due to the sparse sampling and the unknown mutual coupling effect between antennas are considered at the same time, and a novel sparse system model for DOA estimation is formulated. Then, a novel sparse Bayesian learning (SBL)-based method named sparse Bayesian learning with the mutual coupling (SBLMC) is proposed, where an expectation-maximum (EM)-based method is established to estimate all the unknown parameters including the noise variance, the mutual coupling vectors, the off-grid vector and the variance vector of scattering coefficients. Additionally, the prior distributions for all the unknown parameters are theoretically derived. With regard to the DOA estimation performance, the proposed SBLMC method can outperform state-of-the-art methods in the MIMO radar with unknown mutual coupling effect, while keeping the acceptable computational complexity.