Linear Decoders
From Ufldl
(Created page with "In the autoencoder exercises so far, you have been using '''sigmoid decoders''' to reconstruct the input from the features (activations of the hidden layer). That is, the activat...") |
|||
| Line 1: | Line 1: | ||
| - | In the autoencoder exercises so far, you have been using '''sigmoid decoders''' to reconstruct the input from the | + | In the autoencoder exercises so far, you have been using '''sigmoid decoders''' to reconstruct the input from the activations of the hidden units (features). That is, the activation function of the output units has been a sigmoid function. Formally, taking <math>a</math> to be the activations of the hidden units, <math>W^{(2)}, \, b^{(2)}</math> to be the weight and bias terms for the output units, and <math>\hat{x}</math> to be the output of the output units, you have been reconstructing the output as <math>\hat{x} = \sigma(W^{(2)}a + b^{(2)})</math>, where <math>\sigma(x)</math> is the sigmoid function applied to <math>x</math>. |
However, in practice, sigmoid decoders are rarely used because of one major limitation - the limited range of the sigmoid function. Since the range of the sigmoid function is the interval <math>[0, 1]</math>, for datasets in which the input data is not naturally represented in this range, additional pre-processing must be done to scale the data into this range. While it may appear such pre-processing would involve simply scaling the data down linearly into the interval <math>[0, 1]</math>, this does not always work, and the additional pre-processing step is an unnecessary complication. (If you're curious about what other kinds of pre-processing might be necessary, you can look at the pre-processing done for the natural image dataset used in the first sparse autoencoder assignment) | However, in practice, sigmoid decoders are rarely used because of one major limitation - the limited range of the sigmoid function. Since the range of the sigmoid function is the interval <math>[0, 1]</math>, for datasets in which the input data is not naturally represented in this range, additional pre-processing must be done to scale the data into this range. While it may appear such pre-processing would involve simply scaling the data down linearly into the interval <math>[0, 1]</math>, this does not always work, and the additional pre-processing step is an unnecessary complication. (If you're curious about what other kinds of pre-processing might be necessary, you can look at the pre-processing done for the natural image dataset used in the first sparse autoencoder assignment) | ||
