- make-config dim hidden-dim n-layers n-heads n-kv-heads vocab-size seq-len shared-weightsprocedure
Returns a model configuration.
- dim
- embedding dimension
- hidden-dim
- hidden dimension of the feed-forward network
- n-layers
- number of transformer layers
- n-heads
- number of attention heads
- n-kv-heads
- number of key/value heads
- vocab-size
- vocabulary size
- seq-len
- maximum sequence length
- shared-weights
- true when the classifier shares the token embedding table