Exercise:Sparse Coding
From Ufldl
(→Step 3: Iterative optimization) |
(→Step 2: Implement and check sparse coding cost functions) |
||
Line 36: | Line 36: | ||
Once you have implemented these functions, you should check the gradients numerically. | Once you have implemented these functions, you should check the gradients numerically. | ||
- | '''Implementation tip''' - gradient checking the feature cost. One particular point to note is that when checking the gradient for the feature cost, <tt>epsilon</tt> should be set to a larger value, for instance <tt>1e-2</tt> (as has been done for you in the checking code provided), to ensure that checking the gradient numerically makes sense. This is necessary because as <tt>epsilon</tt> becomes smaller, the function <tt>sqrt( | + | '''Implementation tip''' - gradient checking the feature cost. One particular point to note is that when checking the gradient for the feature cost, <tt>epsilon</tt> should be set to a larger value, for instance <tt>1e-2</tt> (as has been done for you in the checking code provided), to ensure that checking the gradient numerically makes sense. This is necessary because as <tt>epsilon</tt> becomes smaller, the function <tt>sqrt(x + epsilon)</tt> becomes "sharper" and more "pointed", making the numerical gradient computed near 0 less and less accurate. To see this, consider what would happen if the numerical gradient was computed by using a point with x less than 0 and a point with x greater than 0 - the computed numerical slope would be wildly inaccurate. |
=== Step 3: Iterative optimization === | === Step 3: Iterative optimization === |