Clustering, multicollinearity, and singular vectors

Let A be a matrix with its pseudo-matrix Adagger and set S=I-AdaggerA. We prove that, after re-ordering the columns of A, the matrix S has a block-diagonal form where each block corresponds to a set of linearly dependent columns. This allows us to identify redundant columns in A. We explore some applications in supervised and unsupervised learning, specially feature selection, clustering, and sensitivity of solutions of least squares solutions.