chickadee » nanograd » layer-norm

layer-norm x gamma beta #!key (epsilon 1e-05)procedure

Normalizes each row of x over its last axis. Each row x_r of length D is mapped to

y_r = gamma * (x_r - mean(x_r)) / sqrt(var(x_r) + epsilon) + beta

where var is the biased variance. gamma and beta are tensors of shape (D). Gradients flow to x, gamma and beta.