IF3270 Pembelajaran Mesin · Recurrent Models Study Tool

Recurrent Neural Networks
& LSTM Cells

Parameters · Activations · Gates · Topology · Unrolled Timeline

Vanilla RNN LSTM Hidden State Cell State Forget Gate Input Gate Output Gate BPTT Bidirectional Layer Norm

References: Hochreiter & Schmidhuber (1997) · Olah (2015) · Goodfellow et al. (2016) Deep Learning · Graves (2013)

01 · Model— params

Activation σtanh

Gate Activationσ

02 · Dimensionsx · h · y

d_x input

d_h hidden

d_y output

T seq length

B batch

Layers

03 · Topology

Direction

Sequencing

Bias

Peephole connections (LSTM) Forget-gate bias = 1.0 Layer normalization Residual h-skip

04 · Regularization

Input dropout0.00

Recurrent dropout0.00

Initialization

Gradient clip1.0

05 · Visual

Diagram view

Density

Accent palette

Show weight matrices Show formulas Show unrolled timeline

ModelRNN

Params— weights

Mem · fwd— MB

FLOPs / step— M

Sequence— steps

Output—

Folded recurrent cell · standard input · x hidden · h gate cell · c

Figure 1.0 — RNN cell after Olah (2015), redrawn for the construction kit.

EQ · EquationsRNN

∑ · Computedlive

W · Weight matrices— tensors

T · Unrolled timelineT = 8

RNN-LSTM-CK · §1.3 h₀, c₀ ← 0 · all values derived live from knob state rev. 06

Recurrent Neural Networks& LSTM Cells