Kafka

In the Ultra Transcenders books

DBX-DEP

Each book explains Kafka in context, with comparison tables and the common traps.

Terms in this definition

• Chat message roles

  Labels on chat messages: instructions go under system, the person's input under user, the model's previous answers under assistant, and results returned by a called tool under tool (or function).

• Agents (classic) API

  First-generation Foundry Agent Service API, based on threads, messages and runs. It is deprecated, replaced by conversations and responses, and retires on 31 March 2027.

• Structured Streaming

  Spark's engine for near real-time data: you write a batch-like query and it processes new data incrementally, recording progress in a checkpoint with exactly-once guarantees. Auto Loader and streaming tables run on it.

• Sink

  Where a stream-processing pipeline sends its output: perhaps a database table, a file, another queue for downstream work, or a live dashboard.

• LEAST

  Pass up to 254 expressions to this T-SQL function and it hands back whichever is smallest. Both Fabric Warehouse and SQL analytics endpoints support it.

• Once flow

  A flow that executes a single time and repeats only if the pipeline is fully refreshed. It's a common choice for a first full load ahead of a change feed, and that refresh becomes dangerous once the original files no longer exist.

Related terms

• Amazon MSK

  Amazon's managed Apache Kafka offering on AWS. In Fabric Real-Time Intelligence it can be added as an eventstream source to pull in its topics.

• Amazon MSK

  Kafka operated for you by AWS. Databricks signs in using IAM or, from Runtime 16.1 onwards, a Unity Catalog service credential.

• Azure HDInsight

  Runs managed clusters for open-source big data frameworks, Hadoop, Spark, Hive and Kafka among them. For new analytics work, Microsoft's current guidance centres on Microsoft Fabric and Azure Databricks.

• CES

  Sends every row change in a stream group's tables to Fabric Eventstream or Azure Event Hubs, each as a CloudEvent. It's in preview for SQL database in Fabric, Managed Instance, Azure SQL Database and SQL Server 2025; delivery over AMQP was deprecated on 15 August 2026, with Kafka the recommended route.

• Custom endpoint

  Lets your own app push events into, or read events out of, an eventstream. It provides connection details for Kafka, AMQP or Event Hubs protocols, secured with either Entra ID or a SAS key, and can act as a source or a destination.

• Eventstream

  Lets you pull live events into Microsoft Fabric from Apache Kafka, Azure IoT Hub, Azure Event Hubs and similar sources, reshape them and pass them on elsewhere, all with no coding; it belongs to Real-Time Intelligence.

• SASL

  Kafka's pluggable authentication layer (Simple Authentication and Security Layer). For Event Hubs, clients use SASL_SSL, picking PLAIN with a connection string or OAUTHBEARER for Microsoft Entra ID.

• SSL

  The older protocol that TLS replaced. You still meet the name in the Kafka setting SASL_SSL and the phrase SSL certificate.