thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Advanced

Pipeline Parallelism

Splitting a model's layers into sequential stages placed on different GPUs, with micro-batches flowing through.

Definition

Pipeline parallelism reduces per-GPU memory and tolerates lower bandwidth than tensor parallelism, but introduces pipeline bubbles that micro-batching minimizes.

Exam tip

Pipeline parallelism splits layers into stages; micro-batching reduces idle pipeline bubbles.

Related terms