PCA and independent component analysis both produce a linear transform of the data. State precisely what each optimises, why PCA cannot recover independent sources in general, and what assumption ICA…

Question

PCA and independent component analysis both produce a linear transform of the data. State precisely what each optimises, why PCA cannot recover independent sources in general, and what assumption ICA needs that PCA does not.

Accepted Answer

PCA finds orthogonal directions of maximal variance and only makes the components uncorrelated, while ICA finds a possibly non-orthogonal basis whose components are statistically independent, which requires at most one source to be Gaussian. Uncorrelatedness is a statement about second moments alone, and independence requires the joint density to factorise, so decorrelation is strictly weaker: two variables can be uncorrelated and strongly dependent, for example through their variances. The reason PCA cannot separate sources is a rotation ambiguity. If the data is whitened, any orthogonal rotation leaves the covariance as the identity, so second-order statistics cannot distinguish among rotations and cannot identify which basis holds the sources. ICA resolves this by using higher-order structure, maximising non-Gaussianity of the projections through kurtosis or negentropy, or equivalently minimising the mutual information among the components. That is exactly why the Gaussian case is excluded: a rotation of independent Gaussians is again independent Gaussians, so the sources are unidentifiable. ICA also leaves the scale and ordering undetermined, since scaling a source and inversely scaling its mixing column changes nothing. On return panels PCA gives variance-ordered factors while ICA aims at the driving shocks, and its output carries no meaningful ordering.