- make-transformer vocab-size max-seq-len d-model n-layers n-heads #!key (d-ff #f) (use-rope? #t) (rope-base 10000.0) (causal? #t) (dtype 'f32) (name Transformer)procedure
Creates the complete model. d-ff is the hidden size of the feed-forward network and defaults to (* 4 d-model). With causal?, each token can attend only to itself and earlier tokens.
(forward model tokens) takes a list or a tensor of token ids and returns the logits, of shape (seq_len, vocab_size). (forward model tokens kv-cache: cache) appends the tokens after those already cached and returns the logits of the last token only, of shape (vocab_size).
The parameters are in this order: the token embedding, the final norm weight, then for each block its two norm weights, its attention parameters (WQ, WK, WV, WO, each a weight and a bias) and its feed-forward parameters (W1, W2, W3, each a weight and a bias).