Softmax Regression

From Ufldl

Jump to: navigation, search
(Cost Function)
(Properties of softmax regression parameterization)
Line 190: Line 190:
p(y^{(i)} = j | x^{(i)} ; \theta)
p(y^{(i)} = j | x^{(i)} ; \theta)
&= \frac{e^{(\theta_j-\psi)^T x^{(i)}}}{\sum_{l=1}^k e^{ (\theta_l-\psi)^T x^{(i)}}}  \\
&= \frac{e^{(\theta_j-\psi)^T x^{(i)}}}{\sum_{l=1}^k e^{ (\theta_l-\psi)^T x^{(i)}}}  \\
-
&= \frac{e^{\theta_j^T x^{(i)}} e^{\psi^Tx}}{\sum_{l=1}^k e^{\theta_l^T} x^{(i)} e^{\psi^Tx}}
+
&= \frac{e^{\theta_j^T x^{(i)}} e^{-\psi^Tx}}{\sum_{l=1}^k e^{\theta_l^T x^{(i)}} e^{-\psi^Tx^{(i)}}}
&= \frac{e^{\theta_j^T x^{(i)}}}{\sum_{l=1}^k e^{ \theta_l^T} x^{(i)}}
&= \frac{e^{\theta_j^T x^{(i)}}}{\sum_{l=1}^k e^{ \theta_l^T} x^{(i)}}
\end{align}
\end{align}
Line 229: Line 229:
make one change to the cost function: Adding weight decay.  This will take care of
make one change to the cost function: Adding weight decay.  This will take care of
the numerical problems associated with softmax regression's overparameterized representation.
the numerical problems associated with softmax regression's overparameterized representation.
-
 
== Weight Decay ==
== Weight Decay ==

Revision as of 07:08, 10 May 2011

Personal tools