chickadee » nanograd » rmsnorm

Root Mean Square layer normalization with batch support.

Input shapes:

• 1-D: (d_model) - standard RMSNorm
• 2-D: (batch_size, d_model) - each row is normalized on its own

Formula: output[i] = (x[i] / RMS(x)) * weight[i], where RMS(x) = sqrt(mean(x^2) + epsilon)

; Single vector
(define x (make-tensor32 (make-f32vector 512) '(512)))
(define gamma (make-tensor32 (make-f32vector 512 1.0) '(512)))
(define normalized (rmsnorm x gamma))

; Batch of vectors
(define batch-x (make-tensor32 (make-f32vector (* 32 512)) '(32 512)))
(define batch-norm (rmsnorm batch-x gamma))  ; Normalized per row