thinQmaster Virtual testing partner Adaptive practice exams that find your weak spots and drill them. Every answer explained. From $69
Enroll

thinQmaster

Find your exam

Five questions free, before you decide.

Step 1 · choose a vendor

In one of our camps? Your practice exams come with the tuition — they are on your study page already.

AI & Machine Learning · Intermediate

LLM-as-a-Judge

Using a language model to grade or compare outputs of another model against a rubric.

Definition

LLM judges score qualities such as correctness, helpfulness, groundedness and tone at scale, often using pairwise comparison or rubric scoring. Judges should be validated against human ratings and checked for position, verbosity and self-preference bias. Amazon Bedrock evaluations and many eval frameworks support this method.

Example

Asking a model to rate each chatbot answer 1-5 for faithfulness to the retrieved context.

Exam tip

LLM-as-a-judge scales evaluation but must be calibrated against human judgments.

Related terms