Independent Component Analysis

 Revision as of 00:47, 18 June 2011 (view source)Cyfoo (Talk | contribs)← Older edit Revision as of 08:22, 19 June 2011 (view source)Jngiam (Talk | contribs) Newer edit → Line 49: Line 49: In practice, the learning rate $\alpha$ is varied using a line-search algorithm to speed up the descent, and the projection step is achieved by setting $W \leftarrow (WW^T)^{-\frac{1}{2}} W$, which can actually be seen as ZCA whitening ([[TODO]] explain how it is like ZCA whitening). In practice, the learning rate $\alpha$ is varied using a line-search algorithm to speed up the descent, and the projection step is achieved by setting $W \leftarrow (WW^T)^{-\frac{1}{2}} W$, which can actually be seen as ZCA whitening ([[TODO]] explain how it is like ZCA whitening). - - == Reconstruction ICA == - - In reconstruction ICA, we drop the constraint that we want an orthonormal basis, replacing it with a reconstruction error term. Hence, the new reconstruction ICA objective is: - :$- \begin{array}{rcl} - {\rm minimize} & \lVert Wx \rVert_1 + \lVert W^TWx \rVert_2^2 \\ - \end{array} -$ - - where $W^T(Wx)$ is the reconstruction of the input from the features (i.e. $W^T$ is supposed to play the role of $W^{-1}$). If you recall the [[Sparse Coding: Autoencoder Interpretation | sparse coding]] objective, - :$- J(A, s) = \lVert As - x \rVert_2^2 + \lambda \lVert s \rVert_1 -$ - you will notice that these two objectives are identical, except that whereas in sparse coding, we are trying to learn a matrix $A$ that maps the feature space to the input space, in reconstruction ICA, we are trying to learn a matrix $W$ that maps the input space to the feature space instead. - - As such, it is obvious that reconstruction ICA does not learn independent or orthonormal components. == Topographic ICA == == Topographic ICA == Just like [[Sparse Coding: Autoencoder Interpretation | sparse coding]], independent component analysis can be modified to give a topographic variant by adding a topographic cost term. Just like [[Sparse Coding: Autoencoder Interpretation | sparse coding]], independent component analysis can be modified to give a topographic variant by adding a topographic cost term.