Cover of Ultra Transcenders Databricks Data Engineer Professional
ULTRA TRANSCENDERS · AN INDEPENDENT STUDY GUIDE

DBX-DEP Databricks Data Engineer Professional

An independent study guide for the Databricks Certified Data Engineer Professional exam · by Tony Rough

Know which Databricks feature fits the workload, why, and what breaks.

  • 17 chapters
  • 20 diagrams
  • 200+ common traps
  • 500+ glossary terms
Amazon.co.ukKindle: coming soonPaperback: coming soon
Amazon.comKindle: coming soonPaperback: coming soon

Due on Amazon in December 2026, in Kindle and paperback editions.

Free glossary of the book's terms

What's inside

The Data Engineer Professional exam isn't about remembering syntax. It's about knowing which Databricks feature fits a workload, why, and what breaks when you pick the tempting alternative.

This independent study guide for the Databricks Certified Data Engineer Professional exam turns the exam outline of October 2026 into the decisions, code patterns and traps that production data engineering on Databricks depends on, written from Databricks' own documentation.

Seventeen chapters, every exam section

Each chapter can be read on its own, and together they cover all nine sections of the exam:

Inside

Understanding, not memorising

There are no exam questions in this book. It explains the knowledge the exam looks for, so you can reason through scenarios you have never seen and bring the same judgement to a real pipeline.

Written by Tony Rough, a cloud architect with over twenty years in IT infrastructure, who designs data platforms for the organisations he works with.

Part of the Ultra Transcenders series from Distilled Press. An independent publication, not affiliated with, sponsored by or endorsed by Databricks, Inc. Databricks, Delta Lake and Unity Catalog are trademarks of Databricks, Inc. or their respective owners.

Is this book for you?

Which certification, which book?

Exam coverage

Every skill area in Databricks's DBX-DEP outline (as of October 9, 2026), and the chapters that cover it.

Skill areaWeightChapters
Section 1: Developing Code for Data Processing using Python and SQL23%1, 2, 3, 4, 5, 6, 10, 11
Section 2: Data Ingestion & Acquisition12%6, 7, 8
Section 3: Data Manipulation12%3, 5, 9
Section 4: Monitoring and Alerting10%10, 12
Section 5: Cost & Performance Optimization15%13, 14
Section 6: Ensuring Data Security and Compliance8%15, 16
Section 7: Data Governance5%15
Section 8: Debugging and Deploying10%10, 11, 14
Section 9: Data Modeling5%17

See every objective and where the book covers it

Chapters

  1. Compute for data engineering
  2. Python projects, libraries and dependencies
  3. UDFs and advanced Spark transformations
  4. Structured Streaming: state and recovery
  5. Lakeflow Pipelines and Auto Loader
  6. Change data capture
  7. Ingesting files and streams
  8. Sharing and federation
  9. VARIANT and AI functions
  10. Lakeflow Jobs
  11. Testing, Git folders and bundles
  12. Monitoring and alerting
  13. Delta table optimisation
  14. Finding performance problems and errors
  15. Unity Catalog access and governance
  16. Privacy, PII and data retention
  17. Data modelling on the lakehouse

Plus an appendix glossary of 500+ terms, each linked to the Databricks documentation, with the same terms explained free online for print readers.

Look inside

Short excerpts from the book, exactly as they appear in it, so you can see how it explains things.

The opening of chapter 1 · Chapter 1: Compute for data engineering

Every workload on Databricks runs on one of three families of compute: serverless compute for notebooks, jobs and pipelines, classic compute that runs in your own cloud account, or a SQL warehouse. The default answer is now serverless, so the useful skill is knowing the exact constraints that push a workload away from it, and the few settings that still matter once it is there. This chapter covers the choice by workload, serverless environments, performance mode, high-memory notebook tasks and the auto-optimisation setting that decides whether a failed task may run twice.

A comparison table · Chapter 1: Compute for data engineering · What serverless jobs compute gives up

Autoscaling and Photon are switched on automatically, and Databricks upgrades the runtime on its own. Cluster creation permission is not needed, so any workspace user can run a serverless job. These are the patterns that have to change.

Classic pattern On serverless What to do instead
RDD APIs, spark.sparkContext, sqlContext Not supported; only Spark Connect APIs DataFrame APIs and spark directly
R, and Scala or R notebooks Not supported Classic compute
Init scripts Not supported Serverless environment
Compute-scoped libraries Not supported Environment dependencies
Maven coordinates Not supported JAR task for jobs; PyPI for notebooks
Compute policies, instance pools Not supported Serverless usage policies for tagging
Most Spark configurations Job runs fail if an unsupported one is set Remove them; serverless tunes most settings
DBFS paths and mounts Access limited; mounts with instance profiles unsupported Unity Catalog volumes and external locations
Hive metastore tables Legacy pattern, not supported Unity Catalog tables, or HMS Federation
VPC peering Not supported Private endpoints, or a firewall allowing serverless traffic
Docker containers Not supported Serverless environments for library needs
Spark UI, Spark logs, compute event logs Not available Query profile and client-side application logs

A common trap · Chapter 1: Compute for data engineering · What serverless jobs compute gives up

Common trap: Serverless jobs inherit the notebook’s 2.5-hour timeout, so a runaway query will be stopped - the default applies only to notebooks. A serverless job has no query execution timeout until spark.databricks.execution.timeout is set.

Sample diagrams

How a watermark, state and the checkpoint work together in a stateful query.
How a watermark, state and the checkpoint work together in a stateful query.
Partitioning sets fixed boundaries, ZORDER arranges data inside them, and liquid clustering replaces both.
Partitioning sets fixed boundaries, ZORDER arranges data inside them, and liquid clustering replaces both.
A grant on a catalog or schema flows to all children, but metastore grants stop at the metastore.
A grant on a catalog or schema flows to all children, but metastore grants stop at the metastore.

Free study notes

Some sections of the book, free to read online:

Study plan

A free week-by-week plan through the book at a 4-week or 8-week pace, with the study notes and exam skill areas for each week.

Open the DBX-DEP study plan

Hear when DBX-DEP comes out

An email when an Ultra Transcenders book is free on Kindle, and when a new book for DBX-DEP comes out. Sign up and you also get the free exam-day checklist: booking, ID, the online check-in and what to expect on the day.

We'll email you to confirm first. A few emails a month at most, no spam, unsubscribe any time. The list is run by Kit; see the privacy notice.

About the author

Tony Rough is a cloud architect with more than twenty years in IT infrastructure, designing Azure platforms for UK organisations at a Microsoft partner. He holds the Azure Solutions Architect Expert, Azure Administrator and Azure Security Engineer certifications and has passed more than thirty Microsoft exams. Ultra Transcenders is the series he wished he'd had.