Model Serving

In the Ultra Transcenders books

DBX-GENAI

Each book explains Model Serving in context, with comparison tables and the common traps.

Terms in this definition

• Databricks

  Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.

• float

  A FLOAT takes up 4 bytes and has single precision. When exactness counts, convert it to DECIMAL, as totals and averages of it may otherwise make a materialized view refresh in full.

• REST

  Short for representational state transfer, the style of HTTP API that Azure services expose.

• API

  Short for application programming interface: a contract that client code calls programmatically, for example a web API secured with tokens or the Files, Images or Responses APIs.

• Serverless compute

  Azure Machine Learning compute provided on demand whenever a job names no compute target. No cluster needs creating or managing, and jobs do not wait in a queue behind one another.

• Legacy

  A label Databricks gives to older agent offerings: their documentation remains so that existing agents can be kept running or migrated, but new agents should avoid them.

• Route table label

  Name for a set of route tables in Virtual WAN hubs, used when choosing where routes propagate; Default, for instance, covers the defaultRouteTable of every hub.

Related terms

• Agent Runtime

  Custom agents on Model Serving are legacy now; their replacement runs agent servers serverlessly on Databricks Apps. Each one has tracing through MLflow, a service principal as its identity, an authenticated endpoint and up to five instances.

• ai_query

  An all-purpose AI Function: it calls a Model Serving endpoint you already have from PySpark or SQL, handling batch and real-time inference, and interprets what comes back.

• Creator identity

  Model Serving records who created an endpoint and uses that identity, permanently, to reach Unity Catalog resources. When the creator loses grants or leaves the workspace, the only fix is to delete the endpoint and build it again.

• Custom LLM serving

  Model Serving GPU endpoints can use this Public Preview feature to host whatever vLLM supports, on a runtime you select, and offer it through an API in OpenAI's format. Don't confuse it with the old Custom LLM agent, since superseded by ai_query.

• Finetuned reranker

  A Beta option for AI Search that trains a reranker on what your index holds and runs it behind Model Serving. The embedding source column has to be the only entry in columns_to_rerank.

• IAM

  Amazon's system for handling roles and permissions. Model Serving endpoints get at resources on AWS via an instance profile, and the role behind that profile must trust serverless compute.

• Model provider services

  The replacement for configuring external models straight on a Model Serving endpoint. Each is a Unity Catalog record of an outside provider, letting Unity Gateway govern its use, track costs and offer it to several workspaces.

• pyfunc

  The flavour MLflow uses to carry arbitrary Python, whether code or a model, into Model Serving. Custom models extend PythonModel, then go through mlflow.pyfunc.log_model.