- bpe-encode text vocab vocab-scoresprocedure
Encodes text as a list of token ids. Each character is first mapped to its token; adjacent tokens are then merged, highest merge score first, while the merged string is in the vocabulary. Exits with an error message when a character has no token.