Data lake

In the Ultra Transcenders books

DP-900

Each book explains Data lake in context, with comparison tables and the common traps.

Terms in this definition

• OVER

  Gives a T-SQL window function its window: PARTITION BY, ORDER BY and, if wanted, a ROWS or RANGE frame. Rankings and running totals can then be worked out while every row is kept.

• DFS

  Short for Distributed File System, the collective label for two role services: DFS Namespaces and DFS Replication.

• LIKE

  Compares strings with a pattern that can contain the % and _ wildcards. Because it only understands character patterns, searching big volumes of text this way is much slower than using full-text search.

• APPLY

  Evaluates a table-valued expression for every row on its left, inside FROM. Think of OUTER APPLY as a left outer join and CROSS APPLY as an inner join.

• Schema

  The middle part of a Unity Catalog name (catalog.schema.table), grouping tables, views, volumes, functions and models inside a catalog. A grant on it covers everything in it now and later, and nothing inside can be reached without USE SCHEMA.

• General-purpose v1

  The older storage account kind (Storage), which lacks access tiers, Archive and premium file shares and retires on 13 October 2026. Converting to ZRS requires first upgrading to GPv2, a one-way change.

• Blob storage

  Azure's object store for unstructured content like images and video; a single block blob can reach roughly 190.7 TiB.

• Hierarchical namespace

  Enabling this option on a GPv2 storage account makes it ADLS Gen2, bringing directories and ACLs.

Related terms

• ADX

  Azure Data Explorer is Microsoft's KQL-based service for analysing large volumes of data. Log Analytics can reach it through the adx() function, though queries and jobs in the Sentinel data lake cannot.

• Azure Storage

  You reach this Azure data storage platform through a storage account; it covers blobs (Data Lake Storage included), file shares, queues and tables.

• Delta Lake

  Table format adding transactions to files in the data lake. In Synapse, Spark pools can write these tables, while serverless SQL pools can only query them.

• DFS endpoint

  Data Lake Storage Gen2 address in the form https://<account>.dfs.core.windows.net. Use this one, not the blob address, when creating OneLake shortcuts or setting up trusted workspace access to an ADLS Gen2 account.

• Fabric Lakehouse

  Combines a data lake with database-style access: Delta tables, files and folders sit in OneLake, and the SQL and Spark engines can both query them. It is one of the item types in Fabric.

• Jupyter notebook

  Analysts hunting or investigating in Microsoft Sentinel can work in these documents, where code sits alongside its output. They are run in Azure Machine Learning, or in Visual Studio Code with the Sentinel extension when the target is the data lake.

• Microsoft Defender for Storage

  Threat detection and malware scanning for Azure Files and Blob Storage, Data Lake Storage Gen2 included, but not for Queue or Table storage. You switch it on for a whole subscription or for individual accounts.

• Microsoft Sentinel MCP server

  Lets AI assistants like Security Copilot or Visual Studio Code ask questions of Sentinel data lake data in everyday language, investigate entities and help triage incidents. Microsoft runs it for you; most of its tools assume you have onboarded to the data lake and hold Security Reader.