chickadee » array-morphisms » var-lift

var-lift fn cell-ranks var ...procedure

Module array-morphisms-grad. The gradient-tracking form of morph-lift. ssa-vjp derives the gradients of lifted computations, and nanograd's am-training-step/ssa uses it. Each cell gets exactly the gradient that its own program would give. An array shared by every cell gets the sum of the cells' gradients, which agrees with adding up per-cell gradients up to rounding. backward! does not differentiate through var-lift.

Measured in f32, a lifted layer tanh(x W + b) with frames of 8 to 64 cells runs its forward pass 6 to 17 times faster than slice-and-stack, and an SSA training step 1.7 to 37 times faster than one program per cell, depending on the BLAS backend.