Softmax Regression
From Ufldl
(→Cost Function) |
(→Properties of softmax regression parameterization) |
||
Line 190: | Line 190: | ||
p(y^{(i)} = j | x^{(i)} ; \theta) | p(y^{(i)} = j | x^{(i)} ; \theta) | ||
&= \frac{e^{(\theta_j-\psi)^T x^{(i)}}}{\sum_{l=1}^k e^{ (\theta_l-\psi)^T x^{(i)}}} \\ | &= \frac{e^{(\theta_j-\psi)^T x^{(i)}}}{\sum_{l=1}^k e^{ (\theta_l-\psi)^T x^{(i)}}} \\ | ||
- | &= \frac{e^{\theta_j^T x^{(i)}} e^{\psi^Tx}}{\sum_{l=1}^k e^{\theta_l^T | + | &= \frac{e^{\theta_j^T x^{(i)}} e^{-\psi^Tx}}{\sum_{l=1}^k e^{\theta_l^T x^{(i)}} e^{-\psi^Tx^{(i)}}} |
&= \frac{e^{\theta_j^T x^{(i)}}}{\sum_{l=1}^k e^{ \theta_l^T} x^{(i)}} | &= \frac{e^{\theta_j^T x^{(i)}}}{\sum_{l=1}^k e^{ \theta_l^T} x^{(i)}} | ||
\end{align} | \end{align} | ||
Line 229: | Line 229: | ||
make one change to the cost function: Adding weight decay. This will take care of | make one change to the cost function: Adding weight decay. This will take care of | ||
the numerical problems associated with softmax regression's overparameterized representation. | the numerical problems associated with softmax regression's overparameterized representation. | ||
- | |||
== Weight Decay == | == Weight Decay == |