Long Short Term Memory Networks Explanation

To solve the problem of Vanishing and Exploding Gradients in a deep Recurrent Neural Network, many variations were developed. One of the most famous of them is the Long Short Term Memory Network(LSTM). In concept, an LSTM recurrent unit tries to “remember” all the past knowledge that the network is seen so far and to “forget” irrelevant data. This is done by introducing different activation function layers called “gates” for different purposes. Each LSTM recurrent unit also maintains a vector called the Internal Cell State which conceptually describes the information that was chosen to be retained by the previous LSTM recurrent unit. A Long Short Term Memory Network consists of four different gates for different purposes as described below:-

1. Forget Gate(f): It determines to what extent to forget the previous data.
2. Input Gate(i): It determines the extent of information to be written onto the Internal Cell State.
3. Input Modulation Gate(g): It is often considered as a sub-part of the input gate and many literatures on LSTM’s do not even mention it and assume it inside the Input gate. It is used to modulate the information that the Input gate will write onto the Internal State Cell by adding non-linearity to the information and making the information Zero-mean. This is done to reduce the learning time as Zero-mean input has faster convergence. Although this gate’s actions are less important than the others and is often treated as a finesse-providing concept, it is good practice to include this gate into the structure of the LSTM unit.
4. Output Gate(o): It determines what output(next Hidden State) to generate from the current Internal Cell State.
   The basic work-flow of a Long Short Term Memory Network is similar to the work-flow of a Recurrent Neural Network with only difference being that the Internal Cell State is also passed forward along with the Hidden State.

Working of an LSTM recurrent unit:

1. Take input the current input, the previous hidden state and the previous internal cell state.
2. Calculate the values of the four different gates by following the below steps:-

• For each gate, calculate the parameterized vectors for the current input and the previous hidden state by element-wise multiplication with the concerned vector with the respective weights for each gate.
• Apply the respective activation function for each gate element-wise on the parameterized vectors. Below given is the list of the gates with the activation function to be applied for the gate.

Input Gate : Sigmoid Function
Forget Gate : Sigmoid Function
Output Gate : Sigmoid Function
Input Modulation Gate : Hyperbolic Tangent Function

3. Calculate the current internal cell state by first calculating the element-wise multiplication vector of the input gate and the input modulation gate, then calculate the element-wise multiplication vector of the forget gate and the previous internal cell state and then adding the two vectors.

4. Calculate the current hidden state by first taking the element-wise hyperbolic tangent of the current internal cell state vector and then performing element wise multiplication with the output gate.

The above stated working is illustrated as below:-

Note that the blue circles denote element-wise multiplication. The weight matrix W contains different weights for the current input vector and the previous hidden state for each gate.

Just like Recurrent Neural Networks, an LSTM network also generates an output at each time step and this output is used to train the network using gradient descent.

The only main difference between the Back-Propagation algorithms of Recurrent Neural Networks and Long Short Term Memory Networks is related to the mathematics of the algorithm.

Let overline{y}_{t} be the predicted output at each time step and y_{t} be the actual output at each time step. Then the error at each time step is given by:-

The total error is thus given by the summation of errors at all time steps.

Similarly, the value frac{partial E}{partial W} can be calculated as the summation of the gradients at each time step.

Using the chain rule and using the fact that overline{y}_{t} is a function of h_{t} and which indeed is a function of c_{t}, the following expression arises:-

Thus the total error gradient is given by the following:-

Note that the gradient equation involves a chain of partial c_{t} for an LSTM Back-Propagation while the gradient equation involves a chain of partial h_{t} for a basic Recurrent Neural Network.

How does LSTM solve the problem of vanishing and exploding gradients?

Recall the expression for c_{t}.

The value of the gradients is controlled by the chain of derivatives starting from frac{partial c_{t}}{partial c_{t-1}}. Expanding this value using the expression for c_{t}:-

For a basic RNN, the term frac{partial h_{t}}{partial h_{t-1}} after a certain time starts to take values either greater than 1 or less than 1 but always in the same range. This is the root cause of the vanishing and exploding gradients problem. In an LSTM, the term frac{partial c_{t}}{partial c_{t-1}} does not have a fixed pattern and can take any positive value at any time step. Thus, it is not guarenteed that for an infinite number of time steps, the term will converge to 0 or diverge completely. If the gradient starts converging towards zero, then the weights of the gates can be adjusted accordingly to bring it closer to 1. Since during the training phase, the network adjusts these weights only, it thus learns when to let the gradient converge to zero and when to preserve it.