Hidden State

The hidden state can be viewed as a compressed summary of past context in RNN-family models. At each time step, it is updated using both the previous state and the current input. This allows the model to preserve continuity across the sequence. However, when its capacity is limited, it may struggle to carry long-range dependencies.