== Sparse Coding == In this exercise, you will implement [[Sparse Coding: Autoencoder Interpretation | sparse coding]] and [[Sparse Coding: Autoencoder Interpretation | topographic sparse coding]] on black-and-white natural images. In the file <tt>[http://ufldl.stanford.edu/wiki/resources/sparse_coding_exercise.zip sparse_coding_exercise.zip]</tt> we have provided some starter code. You should write your code at the places indicated "YOUR CODE HERE" in the files. For this exercise, you will need to modify '''<tt>sparseCodingWeightCost.m</tt>''', '''<tt>sparseCodingFeatureCost.m</tt>''' and '''<tt>sparseCodingExercise.m</tt>'''. === Dependencies === You will need: * <tt>computeNumericalGradient.m</tt> from [[Exercise:Sparse Autoencoder]] * <tt>display_network.m</tt> from [[Exercise:Sparse Autoencoder]] ''If you have not completed the exercise listed above, we strongly suggest you complete it first.'' === Step 0: Initialization === In this step, we initialize some parameters used for the exercise. === Step 1: Sample patches === In this step, we sample some patches from the <tt>IMAGES.mat</tt> dataset comprising 10 black-and-white pre-whitened natural images. === Step 2: Implement and check sparse coding cost functions === In this step, you should implement the two sparse coding cost functions: <ol> <li><tt>sparseCodingWeightCost</tt> in <tt>sparseCodingWeightCost.m</tt>, which is used for optimizing the weight cost given the features <li><tt>sparseCodingFeatureCost</tt> in <tt>sparseCodingFeatureCost.m</tt>, which is used for optimizing the feature cost given the weights </ol> Each of these functions should compute the appropriate cost and gradient. You may wish to implement the non-topographic version of <tt>sparseCodingFeatureCost</tt> first, ignoring the grouping matrix and assuming that none of the features are grouped. You can then extend this to the topographic version later. Alternatively, you may implement the topographic version directly - using the non-topographic version will then involve setting the grouping matrix to the identity matrix. Once you have implemented these functions, you should check the gradients numerically. '''Implementation tip''' - gradient checking the feature cost. One particular point to note is that when checking the gradient for the feature cost, <tt>epsilon</tt> should be set to a larger value, for instance <tt>1e-2</tt> (as has been done for you in the checking code provided), to ensure that checking the gradient numerically makes sense. This is necessary because as <tt>epsilon</tt> becomes smaller, the function <tt>sqrt(x + epsilon)</tt> becomes "sharper" and more "pointed", making the numerical gradient computed near 0 less and less accurate. To see this, consider what would happen if the numerical gradient was computed by using a point with x less than 0 and a point with x greater than 0 - the computed numerical slope would be wildly inaccurate. === Step 3: Iterative optimization === In this step, you will iteratively optimize for the weights and features to learn a basis for the data, as described in the section on [[Sparse Coding: Autoencoder Interpretation | sparse coding]]. Mini-batching and initialization of the features <math>s</math> has already been done for you. However, you need to still need to fill in the analytic solution to the the optimization problem with respect to the weight matrix, given the feature matrix. Once that is done, you should check that your solution is correct using the given checking code, which checks that the gradient at the point determined by your analytic solution is close to 0. Once your solution has been verified, comment out the checking code, and run the iterative optimization code. 200 iterations should take less than 45 minutes to run, and by 100 iterations you should be able to see bases that look like edges, similar to those you learned in [[Exercise:Sparse Autoencoder | the sparse autoencoder exercise]]. For the non-topographic case, these features will not be "ordered", and will look something like the following: [[File:normalSparseCodingFeatures.png]] For the topographic case, the features will be "ordered topographically", and will look something like the following: [[File:topographicSparseCodingFeatures.png]]
Retrieved from "
What links here