Tokenize text
Splits input text into smaller units called tokens using byte-pair encoding (BPE) for the specified model. Returns both the token strings and their corresponding integer token IDs. The text must be between 1 and 65536 characters.
Splits input text into smaller units called tokens using byte-pair encoding (BPE) for the specified model. Returns both the token strings and their corresponding integer token IDs. The text must be between 1 and 65536 characters.