thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Advanced

NCCL

NVIDIA Collective Communications Library: optimized multi-GPU and multi-node collectives such as all-reduce.

Definition

NCCL (pronounced 'nickel') implements all-reduce, all-gather, broadcast and other collectives over NVLink, PCIe, InfiniBand and RoCE. Distributed training frameworks use it to synchronize gradients.

Exam tip

All-reduce via NCCL is how data-parallel training averages gradients across GPUs.

Related terms