thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Intermediate

Tokenizer

The component that converts text to token IDs and back, defined by its vocabulary and splitting rules.

Definition

A tokenizer is trained alongside or before the model and must match it at inference. It also handles special tokens marking message boundaries, roles and end-of-sequence.

Exam tip

Each model family has its own tokenizer, so token counts for the same text differ between models.

Related terms