Fixed points of the EM algorithm and nonnegative rank boundaries

Mixtures of r independent distributions for two discrete random variables can be represented by matrices of nonnegative rank r. Likelihood inference for the model of such joint distributions leads to problems in real algebraic geometry that are addressed here for the first time. We characterize the set of fixed points of the Expectation-Maximization algorithm, and we study the boundary of the space of matrices with nonnegative rank at most 3. Both of these sets correspond to algebraic varieties with many irreducible components.