thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Intermediate

Pretraining

The initial, compute-intensive training of a model on vast unlabeled data to learn general representations.

Definition

LLM pretraining uses self-supervised next-token prediction over trillions of tokens and thousands of GPUs. It produces a base model with broad knowledge, which later stages adapt for specific behavior.

Exam tip

Pretraining from scratch is rarely the right exam answer due to cost; adapt an existing FM instead.

Related terms