thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Advanced

Tensor Parallelism

Splitting individual layers' weight matrices across GPUs so each computes part of every operation.

Definition

Tensor parallelism handles layers too large for one GPU but needs very high-bandwidth, low-latency links, so it is kept within an NVLink domain.

Exam tip

Tensor parallelism stays inside NVLink domains; pipeline parallelism can span nodes.

Related terms