Step-by-Step Hidden State Calculation in a Recurrent Neural Network

Question

Consider a simplified Recurrent Neural Network (RNN) with a single input and a single output. The hidden state is updated using the recurrence:

$$ h_t = \text{ReLU}(W_{ih} \cdot x_t + W_{hh} \cdot h_{t-1}) $$

Assume the following:

• \( x_t = 3 \) for every time step
• \( h_0 = 0 \)
• \( W_{ih} = 0.4 \)
• \( W_{hh} = 0.6 \)
• Activation function: ReLU

Compute the value of the hidden state \( h_4 \) at time \( t = 4 \).

Answer 1

We compute each hidden state step-by-step using

$$ h_t = \text{ReLU}(W_{ih} \cdot x_t + W_{hh} \cdot h_{t-1}). $$

\( h_1 = \text{ReLU}(0.4 \cdot 3 + 0.6 \cdot 0) = 1.2 \)

\( h_2 = \text{ReLU}(0.4 \cdot 3 + 0.6 \cdot 1.2) = 1.92 \)

\( h_3 = \text{ReLU}(0.4 \cdot 3 + 0.6 \cdot 1.92) = 2.352 \)

\( h_4 = \text{ReLU}(0.4 \cdot 3 + 0.6 \cdot 2.352) = 2.6112 \)

Final Answer: \( h_4 = 2.6112 \)