主成分分析

I.e., the data varies much more in the direction <math>\textstyle u_1</math> than <math>\textstyle u_2</math>.  

If <math>\textstyle x</math> has zero mean, then <math>\textstyle \Sigma</math> is exactly the covariance matrix of <math>\textstyle x</math>.  (The symbol "<math>\textstyle \Sigma</math>", pronounced "Sigma", is the standard notation for denoting the covariance matrix.  Unfortunately it looks just like the summation symbol, as in <math>\sum_{i=1}^n i</math>; but these are two different things.)  

Note: If you are interested in seeing a more formal mathematical derivation/justification of this result, see the CS229 (Machine Learning) lecture notes on PCA (link at bottom of this page).  You won't need to do so to follow along this course, however.   

You can use standard numerical linear algebra software to find these eigenvectors (see Implementation Notes).

Here, <math>\textstyle u_1</math> is the principal eigenvector (corresponding to the largest eigenvalue),

The vectors <math>\textstyle u_1</math> and <math>\textstyle u_2</math> in our example form a new basis in which we  

This is the training set rotated into the <math>\textstyle u_1</math>,<math>\textstyle u_2</math> basis. In the general

More generally, if <math>\textstyle x \in \Re^n</math> and we want to reduce it to  

Another way of explaining PCA is that <math>\textstyle x_{\rm rot}</math> is an <math>\textstyle n</math> dimensional

However, since the final <math>\textstyle n-k</math> components of <math>\textstyle \tilde{x}</math> as defined above would

This also explains why we wanted to express our data in the <math>\textstyle u_1, u_2, \ldots, u_n</math> basis:

The final equality above comes from the definition of <math>\textstyle U</math> [[#Example and Mathematical Background|given earlier]].

We are thus using a 1 dimensional approximation to the original dataset.  

To decide how to set <math>\textstyle k</math>, we will usually look at the '''percentage of variance retained'''  

More generally, let <math>\textstyle \lambda_1, \lambda_2, \ldots, \lambda_n</math> be the eigenvalues  

In our simple 2D example above, <math>\textstyle \lambda_1 = 7.29</math>, and <math>\textstyle \lambda_2 = 0.69</math>.  Thus,

A more formal definition of percentage of variance retained is beyond the scope

In the case of images, one common heuristic is to choose <math>\textstyle k</math> so as to retain 99% of

Note: Usually we use images of outdoor scenes with grass, trees, etc., and cut out small (say 16x16) image patches randomly from these to train the algorithm.  But in practice most feature learning algorithms are extremely robust to the exact type of image  it is trained on, so most images taken with a normal camera, so long as they aren't excessively blurry or have strange artifacts, should work.   

When training on natural images, it makes little sense to estimate a separate mean and

In detail, in order for PCA to work well, informally we require that (i) The

So, we won't use variance normalization.  The only normalization we need to

Concretely, if <math>\textstyle x^{(i)} \in \Re^{n}</math> are the (grayscale) intensity values of

Note that the two steps above are done separately for each image <math>\textstyle x^{(i)}</math>,

If you are training your algorithm on images other than natural images (for example, images of handwritten characters, or images of single isolated objects centered against a white background), other types of normalization might be worth considering, and the best choice may be application dependent. But when training on natural images, using the per-image mean normalization method as given in the equations above would be a reasonable default.