- crunch-fused-bodiesprocedure
The canonical bodies of the fused kernels:
Body Formed by the fusion pass for (mul (in 0) (cast f32 (relu-deriv (in 1)))) gradient times relu derivative (mul (in 0) (cast f32 (sigmoid-deriv (in 1)))) gradient times sigmoid derivative (mul (in 0) (cast f32 (tanh-deriv (in 1)))) gradient times tanh derivative (add (in 0) (cast f32 (mul (in 1) (in 2)))) x*y + z (negate (cast f32 (mul (in 0) (in 1)))) negated product (exp (cast f32 (negate (in 0)))) exp(-x) (mul (in 0) (cast f32 (sigmoid (in 0)))) silu(a) (mul (in 0) (cast f32 (mul (in 1) (cast f32 (sigmoid (in 1)))))) b * silu(a), the Llama feed-forward block The f32 kernel rounds where the body says (cast f32 ...), as storing the intermediate value would; the f64 kernel is the same body without casts.