thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Advanced

All-Reduce

A collective operation that combines values (such as gradients) from all processes and distributes the result back to all.

Definition

In data-parallel training, all-reduce sums or averages gradients across GPUs each step. Its efficiency depends on interconnect bandwidth and algorithms such as ring and tree all-reduce; NVIDIA SHARP offloads it into the network switches.

Exam tip

Slow all-reduce usually points to interconnect bandwidth or topology problems in distributed training.

Related terms