chickadee » nanograd » make-batch-norm-2d

Creates a 2D batch normalization layer. Normalizes activations across the batch dimension:

y = γ * (x - μ) / √(σ² + ε) + β

where μ and σ² are computed from the batch (training mode) or from running statistics (evaluation mode).

num-features

number of channels (C)

epsilon

small constant for numerical stability (default 1e-5)

momentum

momentum for updating running statistics (default 0.1)

dtype

'f32 or 'f64 (default 'f32)

name

layer name

Input shapes:

• 3D: (C, H, W) - treated as batch of 1
• 4D: (N, C, H, W) - standard batch

Output shapes: same as input

;; Create batch norm for 64 channels
(define bn (make-batch-norm-2d 64 epsilon: 1e-5 momentum: 0.1))

;; Training mode: uses batch statistics
(set-training-mode! bn #t)
(define normalized (forward bn input))  ; Input shape: (N, 64, H, W)

;; Evaluation mode: uses running statistics
(set-eval-mode! bn)
(define test-normalized (forward bn test-input))  ; Deterministic output

Batch normalization improves training stability and convergence by:

• Reducing internal covariate shift
• Allowing higher learning rates
• Acting as a form of regularization
• Making networks less sensitive to initialization

Key features:

• Learnable scale (gamma) and shift (beta) parameters
• Running mean and variance maintained for evaluation
• Automatic mode switching between training and evaluation
• Numerical stability with epsilon parameter