chickadee » nanograd-components » make-transformer-block

make-transformer-block d-model n-heads d-ff #!key (rope #f) (causal? #f) (dtype 'f32) (name TransformerBlock)procedure
make-multihead-attention d-model n-heads #!key (rope #f) (causal? #f) (dtype 'f32) (name MultiHeadAttention)procedure
make-ffn-block d-model d-ff #!key (dtype 'f32) (name FFN)procedure

Building blocks: one pre-norm block, multi-head self-attention, and the SwiGLU feed-forward network. Note that the block and attention constructors default to causal?: #f.