Built-in LLM judges are predefined scorers running on Databricks-hosted LLMs. The dividing line between them is whether expectations must be supplied, since curated labels cost time to gather.
| Judge | Arguments | Needs ground truth | Evaluates |
|---|---|---|---|
RelevanceToQuery |
inputs, outputs | No | Response directly relevant to the request |
RetrievalRelevance |
inputs, outputs | No | Retrieved context relevant to the request |
Safety |
inputs, outputs | No | Free from harmful, offensive or toxic material |
RetrievalGroundedness |
inputs, outputs | No | Response grounded in the context |
Guidelines |
inputs, outputs | No | Meets stated natural-language criteria |
ToolCallEfficiency |
inputs, outputs | No | Tool calls free of redundancy |
Correctness |
inputs, outputs, expectations | Yes | Response correct against ground truth |
RetrievalSufficiency |
inputs, outputs, expectations | Yes | Context contains all ground truth facts |
ToolCallCorrectness |
inputs, outputs, expectations | Yes | Tool calls and arguments correct |
ExpectationsGuidelines |
inputs, outputs, expectations | No, but needs guidelines in expectations |
Meets per-example criteria |
A sensible pattern follows from the table. A team with no labels can begin with relevance, groundedness and safety, then add Correctness and RetrievalSufficiency as human labels accumulate. The cookbook confirms this order, advising a start with questions alone and adding ground truth over time. Figure 14.1 sorts the judges by what they need.
Reserved keys in expectations connect labels to judges.
| Key | Used by | Holds |
|---|---|---|
expected_facts |
Correctness |
List of facts that should appear |
expected_response |
Correctness |
Exact or similar expected output |
guidelines |
ExpectationsGuidelines |
Natural-language rules for the row |
expected_retrieved_context |
document_recall scorer |
Documents that should be retrieved |
The older cookbook table maps onto this list and adds measurement types: chunk_relevance and groundedness as judges needing no ground truth, context_sufficiency and correctness as judges needing it, and document_recall as a deterministic metric needing ground truth. Token counts and latency_seconds are deterministic. Collect both retrieval and response metrics, since an application can answer poorly with the right context or well despite faulty retrieval.
Judge models can be changed with the model argument in the form <provider>:/<model-name>, for example Correctness(model="databricks:/databricks-gpt-5-mini"). Judges may use third-party services. With cross-Geo processing disabled they stay in the workspace’s Geo and can return a geo-restriction error. With Partner-powered AI features disabled they need a customer-provided model.
Common trap:
RetrievalGroundednessneeds human-labelled answers because it checks facts - it compares the response with the retrieved context, not with ground truth.RetrievalSufficiencyis the retrieval judge that needs expectations.