Tokenize texts

POST 
/v1/tokenize

By giving a text input, generate a tokenized output of token IDs.

To successfully run an inference request, it is mandatory to enter a personal access token (e.g. flp_XXX) value in the Bearer Token field. Refer to the authentication section on our introduction page to learn how to acquire this variable and visit here to generate your token.

Request

Header Parameters

X-Friendli-Team string

ID of team to run requests as (optional parameter).

application/json

Body

model stringrequired

Code of the model to use. See available model list.

prompt stringrequired

Input text prompt to tokenize.

Responses

200

Successfully tokenized the text.

application/json

Schema

Schema

tokens integer[]

A list of token IDs.

Example

Tokenized output.

{
  "tokens": [
    1824,
    349,
    1350,
    1197,
    16107,
    28804,
    13
  ]
}

Loading...