What is the difference between principal component analysis and factor analysis?

What's the difference between factor analysis and feature selection techniques such as,  principle component analysis and latent dirichlet allocation?

• Answer:

  All methods that you mention in your question are unsupervised learning algorithms that can be interpreted as performing maximum likelihood estimation (or in the case of LDA, Bayesian inference) in a probabilistic generative model. A probabilistic generative model describes how observed data was randomly generated, typically involving hidden variables and parameters you want to estimate or infer. The difference between the methods is thus best illustrated by the differences in the generative models they correspond to. In terms of the generative model, principal component analysis and factor analysis differ only slightly. They both assume the following generative procedure for your D dimensional datapoint y first a k-dimensional (k<D), unobserved random vector x is drawn from a standard normal distribution. Thus components of x are independent and identically distributed. Then x is multiplied by a D-by-k constant weight matrix W to give a D-dimensional vector Wx whose coordinates are still Gaussian, but now they may be dependent and non-standard. Your observed data y = Wx + m + e is noise corrupted version of Wx + m where e is random noise, m is a constant mean vector. W and m are parameters that we learn by maximum likelihood or via the EM algorithm. The difference between PCA and FA is only in the assumed distribution of the noise e. In probabilistic PCA components of e are independent and identically distributed Normals, that is they have the same variance across all dimmensions of y, whereas FA allows Gausian noise whose variance is different for each dimension. From a practical perspective the main difference between PCA and FA models are invariance properties: PCA is rotation-invariant, that is, if you rotate your data around it's mean, the factors rotate with your data. FA is scale invariant, that is if you scale your data around it's mean, the factors and dimension-dependent noise variances scale with it. Thus FA is a better model if your different coordinates may have different units, or are scaled arbitrarily. Great papers on the differences and relation between PCA and FA are: Probabilistic PCA http://www.robots.ox.ac.uk/~cvrg/hilary2006/ppca.pdf Unifying Review http://mlg.eng.cam.ac.uk/zoubin/papers/lds.pdf Unsupervised learning http://mlg.eng.cam.ac.uk/zoubin/papers/ul.pdf Latent Dirichlet allocation is a lot more complicated than PCA or FA. The best way to understand the differences is by reading the original paper by David Blei et al, which explains the generative procedure: http://www.cs.princeton.edu/~blei/papers/BleiNgJordan2003.pdf Or look at the topic on Quora.