# Linear Decoders

 Revision as of 07:22, 5 May 2011 (view source)Cyfoo (Talk | contribs)← Older edit Revision as of 02:03, 22 May 2011 (view source)Jngiam (Talk | contribs) Newer edit → Line 1: Line 1: - In the autoencoder exercises so far, you have been using '''sigmoid decoders''' to reconstruct the input from the activations of the hidden units (features). That is, the activation function of the output units has been a sigmoid function. Formally, taking $a$ to be the activations of the hidden units, $W^{(2)}, \, b^{(2)}$ to be the weight and bias terms for the output units, and $\hat{x}$ to be the output of the output units, you have been reconstructing the output as $\hat{x} = \sigma(W^{(2)}a + b^{(2)})$, where $\sigma(x)$ is the sigmoid function applied to $x$. + == Sparse Autoencoder Recap == - However, in practice, sigmoid decoders are rarely used because of one major limitation - the limited range of the sigmoid function. Since the range of the sigmoid function is the interval $[0, 1]$, for datasets in which the input data is not naturally represented in this range, additional pre-processing must be done to scale the data into this range. While it may appear such pre-processing would involve simply scaling the data down linearly into the interval $[0, 1]$, this does not always work, and the additional pre-processing step is an unnecessary complication. (If you're curious about what other kinds of pre-processing might be necessary, you can look at the pre-processing done for the natural image dataset used in the first sparse autoencoder assignment) + In the sparse autoencoder implementation, we had 3 layers of neurons: the input layer, a hidden layer and an output layer. Recall that each neuron (in the output layer) computes the following: - Hence, in practice, '''linear decoders''' are often used instead. For a linear decoder, the activation function of the output unit is simply the identity function. Formally, to reconstruct the input from the features using a linear decoder, we simply set $\hat{x} = W^{(2)}a + b^{(2)}$ instead, without applying the sigmoid function. Now the reconstructed output $\hat{x}$ is a linear function of the activations of the hidden units, which means that by varying $W$, each output unit $\hat{x}$ can be made to produce any activation in $(-\infty, \infty)$. This allows us to train the sparse autoencoder on any input that takes on real values without any additional pre-processing. (Note that the hidden units are '''still sigmoid units''', that is, $a = \sigma(W^{(1)}*x + b^{(1)})$, where $x$ is the input, and $W^{(1)}$ and $b^{(1)}$ are the weight and bias terms for the hidden units) + $+ z^{(3)} &= W^{(2)} a^{(2)} + b^{(2)} \\ + a^{(3)} &= f(z^{(3)}) +$ + + where $a^{(3)}$ is the reconstruction of the input (layer $a^{(1)}$). + + Notice that due to the choice of the sigmoid function for $f(z^{(3)})$ we need to constrain the inputs to be in the range [0,1]. + + While some datasets like MNIST fit well into this criteria, this hard constraint can sometimes be awkward to satisfy. For example, if one uses PCA whitening, the input is no longer constrained to [0,1] and it is not clear what kind of scaling is appropriate to fit the data into the constrained range. + + == Linear Decoder == + + One easy fix for the fore-mentioned problem is to use a ''linear-decoder'', that is, we set $a^{(3)} &= f(z^{(3)})$. + + For a linear decoder, the activation function of the output unit is effectively the identity function. Formally, to reconstruct the input from the features using a linear decoder, we simply set $\hat{x} = a^{(3)} = z^{(3)} = W^{(2)}a + b^{(2)}$ instead, without applying the sigmoid function. Now the reconstructed output $\hat{x}$ is a linear function of the activations of the hidden units, which means that by varying $W$, each output unit $\hat{x}$ can be made to produce any activation without the previous constraints. This allows us to train the sparse autoencoder on any input that takes on real values without any additional pre-processing. (Note that the hidden units are '''still sigmoid units''', that is, $a = \sigma(W^{(1)}*x + b^{(1)})$, where $x$ is the input, and $W^{(1)}$ and $b^{(1)}$ are the weight and bias terms for the hidden units) Of course, now that we have changed the activation function of the output units, the gradients of the output units must be changed accordingly. Recall that for each output unit, we set the error as follows: Of course, now that we have changed the activation function of the output units, the gradients of the output units must be changed accordingly. Recall that for each output unit, we set the error as follows: