thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Intermediate

Vanishing Gradient

A training problem where gradients shrink toward zero in early layers, so they learn very slowly or not at all.

Definition

Vanishing gradients occur when many small derivatives are multiplied through deep networks or long sequences, especially with sigmoid or tanh activations. Remedies include ReLU-family activations, residual connections, normalization layers, careful initialization and LSTM/GRU gating.

Exam tip

Residual connections and ReLU are the classic fixes for vanishing gradients.

Related terms