Tokenisation

In the Ultra Transcenders books

DBX-DEP

Each book explains Tokenisation in context, with comparison tables and the common traps.

Terms in this definition

• Databricks

  Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.

• FUNCTION

  Registered models count as functions in Unity Catalog, so this securable type governs them too: access is given through GRANT ... ON FUNCTION, and REST calls pass FUNCTION as securable_type.

• AES

  Short for Advanced Encryption Standard, a symmetric cipher. With TDE, the RSA TDE protector wraps an AES-256 data encryption key.

• LOOKUP

  Inside a visual calculation, this DAX function retrieves a value from the visual matrix using whatever filters you supply, and works out any you omit from the context it is evaluated in.

• Event

  Table in Log Analytics where entries from Windows event logs are kept.

• VALUES

  Returns in DAX the distinct column values, or table rows, still visible after filters are applied, sometimes with an extra blank entry. CALCULATE often takes the result as a table filter.

• Index field attributes

  Settings applied to each field in an Azure AI Search index: searchable for full text, retrievable to return it, filterable for exact-match $filter, sortable, facetable for counts, and key for the unique document ID.

• TURN

  If a direct link can't be made, RDP Shortpath for Windows 365 relays UDP traffic via Microsoft servers on port 3478 instead.

Related terms

• Search analyzer

  Component in Azure AI Search that breaks searchable string fields into tokens; it may be standard Lucene, language-specific, built-in or custom. It affects tokenisation rather than meaning.

• Tokenization

  At the start of a transformer's pipeline, input text is cut into tokens and every token is given its ID from the vocabulary; this step is tokenization.