thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Intermediate

Data Parallelism

Distributed training where each GPU holds a full model copy and processes a different slice of each batch.

Definition

Gradients are averaged across GPUs with all-reduce every step. It scales easily but requires the whole model to fit on each GPU, which FSDP and ZeRO relax by sharding states.

Exam tip

If the model fits on one GPU, start with data parallelism.

Related terms