thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Foundational

Tokenization (NLP)

Splitting text into tokens that a model can convert to numeric IDs.

Definition

Modern LLMs use subword tokenization such as byte-pair encoding, WordPiece or SentencePiece, balancing vocabulary size against sequence length. Different models use different tokenizers, so the same text can yield different token counts.

Exam tip

Token counts are model-specific; always count with the target model's tokenizer.

Related terms