thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Intermediate

Attention Mechanism

A technique that lets a model weight the relevance of different input elements when producing each output.

Definition

Attention computes similarity scores between a query and a set of keys, normalizes them with softmax, and uses them to take a weighted sum of values. It lets models focus on the most relevant context regardless of distance in the sequence.

Exam tip

Attention lets a model weigh relevant tokens regardless of distance, solving the long-range dependency problem of RNNs.

Related terms