Evaluation and monitoring phases compared

The two phases share machinery but answer different questions at different times. The older cookbook pages say evaluation happens during development and monitoring in production, with similar fundamental components.

Name changes are common here. The cookbook pages still say Agent Evaluation, its Review App and metric names such as chunk_relevance, whereas the current MLflow 3 pages describe mlflow.genai.evaluate(), scorers, judges and the MLflow Review App. The exam guide’s Agent Monitoring appears on the current pages as production monitoring.

Neither phase replaces the other: monitoring samples whatever users happen to send, so it cannot compare a candidate version with a baseline on the same inputs. The core idea carries over from the cookbook. Classical ML has answers that are deterministically right or wrong; here there can be many right answers, some more right than others. That calls for human feedback for confidence and LLM-judged metrics for scale, which is the structure the rest of this chapter follows.

Common trap: Evaluation and monitoring need separate sets of metrics - the same scorer can run in both, and consistent evaluation across the lifecycle is the purpose of the design. The real constraints are that monitoring has no expectations and that custom scorers must be serialisable.

Get the whole book

This note is one section of Ultra Transcenders DBX-GENAI: Databricks Generative AI Engineer Associate, an independent study guide that explains every topic the exam covers by technology, with comparison tables, diagrams and the common traps, plus a glossary linked to the Databricks documentation.

About the book · DBX-GENAI terms in the glossary · All DBX-GENAI study notes