thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Intermediate

Inference Throughput

The amount of work an inference system completes per unit time, such as tokens or requests per second.

Definition

Throughput and latency trade off: larger batches raise total tokens per second but can increase per-request latency. Serving engines use continuous batching and efficient KV-cache management to maximize throughput.

Exam tip

Batch workloads optimize throughput; chat workloads optimize latency.

Related terms