thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Advanced

Gradient Accumulation

Summing gradients over several mini-batches before updating weights to simulate a larger batch.

Definition

Gradient accumulation lets training use an effective batch size larger than GPU memory permits, trading extra time for memory. Effective batch size equals per-device batch times accumulation steps times number of devices.

Exam tip

Use gradient accumulation when the desired batch does not fit in GPU memory.

Related terms