- make-kv-cache n-layers n-heads max-seq-len head-size #!key (dtype 'f32)procedure
- kv-cache-length cacheprocedure
- clear-cache! cacheprocedure
make-kv-cache creates a cache for the keys and values of one text. kv-cache-length returns the number of cached positions, and clear-cache! empties the cache. The cache stores keys and values as constants, so use it for generation only, not for training.