thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Advanced

KV Cache

Stored attention keys and values for previous tokens, reused so each new token does not recompute them.

Definition

The KV cache makes autoregressive generation efficient but consumes GPU memory proportional to sequence length, layers and batch size, often limiting concurrency. PagedAttention, GQA, quantized caches and KV offloading reduce its footprint.

Exam tip

Long contexts and many concurrent users are limited by KV cache memory.

Related terms