Tokenize texts
POST/v1/tokenize
By giving a text input, generate a tokenized output of token IDs.
To successfully run an inference request, it is mandatory to enter a personal access token (e.g. flp_XXX) value in the Bearer Token field. Refer to the authentication section on our introduction page to learn how to acquire this variable and visit here to generate your token.
Request
Header Parameters
X-Friendli-Team string
ID of team to run requests as (optional parameter).
- application/json
Body
model stringrequired
Code of the model to use. See available model list.
prompt stringrequired
Input text prompt to tokenize.
Responses
- 200
Successfully tokenized the text.
- application/json
- Schema
- Example
Schema
tokens integer[]
A list of token IDs.
Tokenized output.
{
"tokens": [
1824,
349,
1350,
1197,
16107,
28804,
13
]
}
Loading...