thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Intermediate

Self-Attention

Attention in which each token in a sequence attends to every other token in the same sequence.

Definition

Self-attention builds context-aware token representations, for example resolving what 'it' refers to in a sentence. Its compute and memory grow quadratically with sequence length, which drives context-window cost and techniques such as FlashAttention.

Exam tip

Self-attention cost scales quadratically (O(n^2)) with sequence length.

Related terms