chickadee » array-morphisms-crunch » crunch-fused-bodies

crunch-fused-bodiesprocedure

The canonical bodies of the fused kernels:

BodyFormed by the fusion pass for
(mul (in 0) (cast f32 (relu-deriv (in 1))))gradient times relu derivative
(mul (in 0) (cast f32 (sigmoid-deriv (in 1))))gradient times sigmoid derivative
(mul (in 0) (cast f32 (tanh-deriv (in 1))))gradient times tanh derivative
(add (in 0) (cast f32 (mul (in 1) (in 2))))x*y + z
(negate (cast f32 (mul (in 0) (in 1))))negated product
(exp (cast f32 (negate (in 0))))exp(-x)
(mul (in 0) (cast f32 (sigmoid (in 0))))silu(a)
(mul (in 0) (cast f32 (mul (in 1) (cast f32 (sigmoid (in 1))))))b * silu(a), the Llama feed-forward block

The f32 kernel rounds where the body says (cast f32 ...), as storing the intermediate value would; the f64 kernel is the same body without casts.