thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Advanced

Fully Sharded Data Parallel (FSDP)

A PyTorch training strategy that shards parameters, gradients and optimizer states across data-parallel GPUs.

Definition

FSDP, similar to DeepSpeed ZeRO stage 3, gathers parameters only when needed for a layer's computation, letting much larger models train with data parallelism.

Exam tip

Choose FSDP when the model is too large for one GPU under plain data parallelism.

Related terms