chickadee » nanograd » rmsnorm

Root Mean Square Layer Normalization with batch support.

Input shapes:

• 1D: (d_model,) - standard RMSNorm
• 2D: (batch_size, d_model) - RMSNorm applied to each batch element independently

Formula: output[i] = (x[i] / RMS(x)) * weight[i] where RMS(x) = sqrt(mean(x^2) + epsilon)

; Single vector
(define x (make-tensor32 (make-f32vector 512) '(512)))
(define gamma (make-tensor32 (make-f32vector 512 1.0) '(512)))
(define normalized (rmsnorm x gamma))

; Batch of vectors
(define batch-x (make-tensor32 (make-f32vector (* 32 512)) '(32 512)))
(define batch-norm (rmsnorm batch-x gamma))  ; Normalized per batch element