thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Foundational

Token

The basic unit of text an LLM processes, typically a word, subword, character or punctuation mark.

Definition

Text is split into tokens by a tokenizer and mapped to integer IDs. In English, one token averages roughly four characters or three-quarters of a word. Model pricing, rate limits and context windows are all measured in tokens.

Example

'unbelievable' might tokenize as 'un', 'believ', 'able'.

Exam tip

Costs and context limits count input plus output tokens, not words or characters.

Related terms