Exercise:Independent Component Analysis

From Ufldl

Jump to: navigation, search
(Step 4: Optimization)
(Backtracking line search)
 
Line 54: Line 54:
The backtracking line search used in the exercise is based off that in [http://www.stanford.edu/~boyd/cvxbook/ Convex Optimization by Boyd and Vandenbergh]. In the backtracking line search, given a descent direction <math>\vec{u}</math> (in this exercise we use <math>\vec{u} = -\nabla f(\vec{x})</math>), we want to find a good step size <math>t</math> that gives us a steep descent. The general idea is to use a linear approximation (the first order Taylor approximation) to the function <math>f</math> at the current point <math>\vec{x}</math>, and to search for a step size <math>t</math> such that we can decrease the function's value by more than <math>\alpha</math> times the decrease predicted by the linear approximation (<math>\alpha \in (0, 0.5)</math>. For more details, you may wish to consult [http://www.stanford.edu/~boyd/cvxbook/ the book].
The backtracking line search used in the exercise is based off that in [http://www.stanford.edu/~boyd/cvxbook/ Convex Optimization by Boyd and Vandenbergh]. In the backtracking line search, given a descent direction <math>\vec{u}</math> (in this exercise we use <math>\vec{u} = -\nabla f(\vec{x})</math>), we want to find a good step size <math>t</math> that gives us a steep descent. The general idea is to use a linear approximation (the first order Taylor approximation) to the function <math>f</math> at the current point <math>\vec{x}</math>, and to search for a step size <math>t</math> such that we can decrease the function's value by more than <math>\alpha</math> times the decrease predicted by the linear approximation (<math>\alpha \in (0, 0.5)</math>. For more details, you may wish to consult [http://www.stanford.edu/~boyd/cvxbook/ the book].
 +
 +
However, it is not necessary to use the backtracking line search here. Gradient descent with a small step size, or backtracking to a step size so that the objective decreases is sufficient for this exercise.

Latest revision as of 04:31, 4 October 2011

Personal tools