Exercise:Self-Taught Learning
From Ufldl
for
Exercise:Self-Taught Learning
Jump to:
navigation
,
search
===Overview=== In this exercise, we will use the self-taught learning paradigm with the sparse autoencoder and softmax classifier to build a classifier for handwritten digits. In this context, the self-taught learning paradigm can be used as follows. First, the sparse autoencoder is trained on a unlabeled training dataset of images of handwritten digits. This produces feature detectors that correspond to the activation pattern for each hidden unit. The features learned are expected to be pen strokes to resemble segments of digits. The sparse autoencoder is then used to trainsform the raw labeled training dataset into representation of features, which are then used to train train the softmax classifier. For each example in the the labeled training dataset, <math>\textstyle x^{(k)}</math>, forward propogation is used to is used to obtain the activation of the hidden units <math>\textstyle a^{(k)}</math>. The data point <math>\textstyle a^{(k)}</math> and its corresponding label is then used to train the softmax classifier. Finally, the test dataset is also transformed to hidden unit activations and passed to the softmax classifier to obtain predictions. To demonstrate the effectiveness of this method, we will train the sparse autoencoder on an unlabeled data set comprised out of all the digits 0 to 9, and then test them on the digits 5 to 9. The purpose of this is to demonstrate that self-taught learning can be surprisingly effective in improving results even if some data set items do not fall within the classes of our classification task. === Support Code/Data === The following additional files are required for this exercise: * [http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz MNIST Dataset (Training Images)] * [http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz MNIST Dataset (Training Labels)] * [[Using the MNIST Dataset | Support functions for loading MNIST in Matlab ]] This exercise has dependencies on the previous exercises: * [[Exercise:Sparse Autoencoder]] * [[Exercise:Vectorization]] * [[Exercise:Softmax Regression]] ===Step One: Generate the input and test data sets=== Download and decompress <tt>[http://ufldl.stanford.edu/wiki/resources/stl_exercise.zip stl_exercise.zip]</tt>, which contains starter code for this exercise. Additionally, you will need to download the datasets from the MNIST Handwritten Digit Database for this project. To separate the the cases, you will need to fill in <tt>filterData.m</tt>, which should return a subset of the data set containing examples with labels within a certain bound. After implementing this function, run this step. To test your function, the script will output the number of examples in the supervised training and the STL training data set, which should be 60000 and 30596 respectively. ===Step Two: Train the sparse autoencoder=== Next we will train the input data set on the sparse autoencoder, using the same <tt>sparseAutoencoderCost.m</tt> function from the previous assignments. (Use the frameworks from previous assignments to ensure that your code is working and vectorized, as no testing facilities are provided in this assignment.) After doing so, running the training step should take about half an hour (on a reasonably fast computer). When it is completed, a visualization of pen strokes should be displayed. Hint: This step takes a very long time, so you might want to avoid running it on subsequent trials! To do so, uncomment the lines containing the function <tt>save(...);</tt>, or run the save function from the command line. In future trials, you may skip the training set and load the save weights from disk. [[File:selfTaughtFeatures.png]] ===Step Three: Training the logistic regression model=== After the sparse autoencoder is trained, we can use it to detect pen strokes in images. To do so, you will need to be able to save the sparse autoencoder function to output its hidden unit activation. Fill in <tt>feedForwardSparseAutoencoder.m</tt> to produce a matrix whose columns correspond to activation of the hidden layer for each example i.e. the vector <math>a^{(2)}</math> corresponding to activation of layer <math>L_{2}</math>. After doing so, this step will use your modified function to convert the raw image data to feature unit activations. It will then train the softmax regression model on the hidden unit activation and labels with <tt>trainSoftmax.m</tt>. ===Step Four: Training and testing the logistic regression model=== After completing these steps, running the entire script in <tt>trainSelfTaught.m</tt> will use your sparse autoencoder to train the logistic model, then measure how well this system performs on the test set. Statistics about the model will be displayed afterwards. If you've done all the steps correctly, you should get an accuracy of about 98% percent. [[Category:Exercises]]
Template:STL
(
view source
)
Return to
Exercise:Self-Taught Learning
.
Views
Page
Discussion
View source
History
Personal tools
Log in
ufldl resources
UFLDL Tutorial
Recommended Readings
wiki
Main page
Recent changes
Random page
Help
Search
Toolbox
What links here
Related changes
Special pages