thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Advanced

Multi-Head Attention (MHA)

Running several attention operations in parallel so the model can attend to different relationships at once.

Definition

Each head has its own query, key and value projections and may specialize in syntax, coreference or other patterns. Outputs are concatenated and projected. Grouped-query attention shares key/value heads across query heads to shrink the KV cache.

Exam tip

Multiple heads let the model capture different relationships in parallel; outputs are concatenated and projected.

Related terms