Exercise:Vectorization

From Ufldl

Jump to: navigation, search
 
Line 1: Line 1:
== Vectorization ==
== Vectorization ==
-
In the previous problem set, we implemented a sparse autoencoder for patches taken from natural images. In this problem set, you will adapt the sparse autoencoder to work on images of handwritten digits. You will be given a working but unvectorized implementation, and your task will be to vectorize a key step to improve its performance.
+
In the previous problem set, we implemented a sparse autoencoder for patches taken from natural images. In this problem set, you will vectorize your code to make it run much faster, and further adapt your sparse autoencoder to work on images of handwritten digits. Your network for learning from handwritten digits will be much larger than the one you'd trained on the natural images, and so using the original implementation would have been painfully slow.  But with a vectorized implementation of the autoencoder, you will be able to get this to run in a reasonable amount of computation time.  
-
In the file <tt>vec_assign.zip</tt>, you will find MATLAB code implementing a sparse autoencoder. To run the code, you will need to download an additional data set from the [http://yann.lecun.com/exdb/mnist/|MNIST handwritten digit database]. Download the file <tt>train-images-idx3-ubyte.gz</tt> and decompress it to the <tt>MNIST/</tt> folder in the project path. You should then be able to run <tt>train.m</tt> and obtain some preliminary results.
+
=== Support Code/Data ===
-
=== MNIST ===
+
The following additional files are required for this exercise:
 +
* [http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz MNIST Dataset (Training Images)]
 +
* [http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz MNIST Dataset (Training Labels)]
 +
* [[Using the MNIST Dataset | Support functions for loading MNIST in Matlab ]]
-
Use the following parameters for the MNIST dataset:
+
=== Step 1: Vectorize your Sparse Autoencoder Implementation ===
-
  patchSize: 28x28 patches
+
Using the ideas from [[Vectorization]] and [[Neural Network Vectorization]], vectorize your implementation of <tt>sparseAutoencoderCost.m</tt>. In our implementation, we were able to remove all for-loops with the use of matrix operations and <tt>repmat</tt>. (If you want to play with more advanced vectorization ideas, also type <tt>help bsxfun</tt>. The <tt>bsxfun</tt> function provides an alternative to <tt>repmat</tt> for some of the vectorization steps, but is not necessary for this exercise).  A vectorized version of our sparse autoencoder code ran in under one minute on a fast computer (for learning 25 features from 10000 8x8 image patches).
 +
 
 +
(Note that you do not need to vectorize the code in the other files.)
 +
 
 +
=== Step 2: Learn features for handwritten digits ===
 +
 
 +
Now that you have vectorized the code, it is easy to learn larger sets of features on medium sized images. In this part of the exercise, you will use your sparse autoencoder to learn features for handwritten digits from the MNIST dataset. 
 +
 
 +
The MNIST data is available at [http://yann.lecun.com/exdb/mnist/]. Download the file <tt>train-images-idx3-ubyte.gz</tt> and decompress it. After obtaining the source images, you should use [[Using the MNIST Dataset | helper functions that we provide]] to load the data into Matlab as matrices.  While the helper functions that we provide will load both the input examples <math>x</math> and the class labels <math>y</math>, for this assignment, you will only need the input examples <math>x</math> since the sparse autoencoder is an ''unsupervised'' learning algorithm.  (In a later assignment, we will use the labels <math>y</math> as well.)
 +
 
 +
The following set of parameters worked well for us to learn good features on the MNIST dataset:
 +
 
 +
visibleSize = 28*28
 +
hiddenSize = 196
  sparsityParam = 0.1
  sparsityParam = 0.1
  lambda = 3e-3
  lambda = 3e-3
  beta = 3
  beta = 3
-
  normalizeData: linear scaling (patches = patches / 255)
+
  patches = first 10000 images from the MNIST dataset
-
The autoencoder should learn pen strokes as features. These features should start to become obvious after 400 iterations of minFunc, which takes around 20 - 25 minutes on the Corn cluster. Visualised, the features should look like in the following image:
+
After 400 iterations of updates using minFunc, your autoencoder should have learned features that resemble pen strokes. In other words, this has learned to represent handwritten characters in terms of what pen strokes appear in an image.  Our implementation takes around 15-20 minutes on a fast machine. Visualized, the features should look like the following image:  
-
[[File:MNIST-false-0.1-3e-3-3-linear.png]]
+
[[File:mnistVectorizationEx.png|400px]]
If your parameters are improperly tuned, or if your implementation of the autoencoder is buggy, you may get one of the following images instead:
If your parameters are improperly tuned, or if your implementation of the autoencoder is buggy, you may get one of the following images instead:
Line 25: Line 41:
</table>
</table>
-
If your image looks like one of the above images, check your code and parameters again. In particular, templates of digits are not very useful as features, since they do not generalise very well to digits written differently.
+
If your image looks like one of the above images, check your code and parameters again. Learning these features are a prelude to the later exercises, where we shall see how they will be useful for classification.
-
=== Natural images ===
+
<!-- === Natural images ===
Use the following parameters for the natural images dataset:
Use the following parameters for the natural images dataset:
-
  patchSize: 14x14 patches
+
  visibleSize = 14*14;
-
  sparsityParam = [[TODO]]
+
hiddenSize = 196;
-
  lambda = [[TODO]]
+
  sparsityParam = 0.035;
-
  beta = [[TODO]]
+
  lambda = 0.0003; 
-
normalizeData: [[TODO]]
+
  beta = 5;
 +
 
 +
As with the first problem, the autoencoder should learn edge features. Your code should run in under 10 minutes on a reasonably fast machine. If it takes significantly longer, check your code and ensure that it is vectorized.
 +
 
 +
[[Category:Exercises]] -->
-
[[TODO]]
 
-
[[Category:Exercises]] [[Category:TODO]]
+
{{Vectorized Implementation}}

Latest revision as of 11:00, 26 May 2011

Personal tools