- bpe-encode text vocab vocab-scoresprocedure
Tokenize text using Byte-Pair Encoding.
- text
- Input text string
- vocab
- List of vocabulary strings
- vocab-scores
- List of BPE merge scores
Returns list of token IDs
Tokenize text using Byte-Pair Encoding.
Returns list of token IDs