- tokenizer-init config portprocedure
Reads the tokenizer file from port and returns three values: the list of token strings (decoded as UTF-8), the list of their merge scores, and the maximum token length.
Reads the tokenizer file from port and returns three values: the list of token strings (decoded as UTF-8), the list of their merge scores, and the maximum token length.