thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Advanced

Continuous Batching

An LLM serving technique that adds and removes requests from the running batch at each generation step.

Definition

Also called in-flight batching, it avoids waiting for an entire batch to finish, keeping GPUs busy and greatly improving throughput for variable-length outputs.

Exam tip

Continuous batching raises throughput for variable-length LLM outputs compared with static batching.

Related terms