- make-config dim hidden-dim n-layers n-heads n-kv-heads vocab-size seq-len shared-weightsprocedure
Creates a new configuration object.
- dim
- Model embedding dimension
- hidden-dim
- FFN hidden layer dimension
- n-layers
- Number of transformer layers
- n-heads
- Number of attention heads
- n-kv-heads
- Number of key-value heads
- vocab-size
- Vocabulary size
- seq-len
- Maximum sequence length
- shared-weights
- Whether to share input/output embeddings