Implement data engineering solutions using Azure Databricks

Explore Azure Databricks

Explore Azure Databricks

• Get started with Azure Databricks
• Identify Azure Databricks workloads
• Understand key concepts
• Data governance using Unity Catalog and Microsoft Purview
• Exercise - Explore Azure Databricks
• Module assessment

Understand Azure Databricks architecture

Learn about Azure Databricks architecture, including the account hierarchy, control and compute planes, and different storage options for managing data with Unity Catalog.

• Understand Azure Databricks architecture
• Understand Unity Catalog managed storage
• Understand external storage
• Understand default storage
• Module assessment

Understand Azure Databricks Integrations

Learn how Azure Databricks integrates with Microsoft tools and services including Fabric, Power BI, VS Code, Power Platform, Copilot Studio, Purview, and Foundry to enable comprehensive data and AI solutions.

• Understand integration with Microsoft Fabric
• Understand integration with Power BI
• Understand integration with VS Code
• Understand integration with Power Platform
• Understand integration with Copilot Studio
• Understand integration with Microsoft Purview
• Understand integration with Microsoft Foundry
• Module assessment

Select and Configure Compute in Azure Databricks

Learn how to select and configure appropriate compute resources in Azure Databricks including serverless, classic compute, SQL warehouses, and job clusters. Master performance tuning, access control, and library management.

• Choose an appropriate compute type
• Configure compute performance
• Configure compute features
• Install libraries for compute
• Configure compute access
• Exercise - Select and Configure Compute in Azure Databricks
• Module assessment

Create and organize objects in Unity Catalog

Learn how to create and organize catalogs, schemas, tables, views, and volumes in Unity Catalog to build a comprehensive data governance framework with proper naming conventions and AI/BI integration.

• Apply naming conventions
• Create catalog
• Create schema
• Create tables and views
• Create volumes
• Implement DDL operations
• Implement foreign catalog
• Configure AI/BI Genie instructions
• Exercise - Create and Organize Objects in Unity Catalog

Secure Unity Catalog objects

Learn how to implement comprehensive security for Unity Catalog objects in Azure Databricks, including access control, fine-grained permissions, credential management, and authentication strategies.

• Understand query lifecycle
• Implement access control strategies
• Understand fine-grained access control
• Implement row filtering and column masking
• Access Azure Key Vault secrets
• Authenticate data access with service principals
• Authenticate resource access with managed identities
• Exercise - Secure Unity Catalog Objects
• Module assessment

Govern Unity Catalog objects

Learn how to govern data and AI assets in Azure Databricks using Unity Catalog, including access control, data lineage, audit logging, and secure sharing.

• Create and preserve table definitions
• Configure ABAC with tags and policies
• Apply data retention policies
• Set up and manage data lineage
• Configure audit logging
• Design secure Delta Sharing strategy
• Exercise - Govern Unity Catalog Objects
• Module assessment

Design and implement data modeling with Azure Databricks

Learn how to design and implement data modeling strategies in Azure Databricks with Unity Catalog, including ingestion patterns, table formats, partitioning, slowly changing dimensions, and clustering strategies.

• Design ingestion logic and data source configuration
• Choose a data ingestion tool
• Choose a data table format
• Design and implement a data partitioning scheme
• Choose a slowly changing dimension (SCD) type
• Implement a slowly changing dimension (SCD) type 2
• Design and implement a temporal (history) table to record changes over time
• Choose granularity on a column or table based on requirements
• Choose managed vs unmanaged tables
• Design and implement a clustering strategy
• Exercise - Design and Implement Data Modeling with Azure Databricks

Ingest data into Unity Catalog

Learn how to ingest data from diverse sources into Unity Catalog tables in Azure Databricks using managed connectors, notebooks, SQL commands, streaming, and declarative pipelines.

• Ingest data with Lakeflow Connect
• Ingest data with notebooks
• Ingest data with SQL methods
• Ingest data with CDC feed
• Ingest data with Spark Structured Streaming
• Ingest data with Auto Loader
• Ingest data with Lakeflow Spark Declarative Pipelines
• Exercise - Ingest Data into Unity Catalog
• Module assessment

Cleanse, transform, and load data into Unity Catalog

Learn how to cleanse, transform, and load data into Unity Catalog tables in Azure Databricks by profiling data, handling duplicates and nulls, applying transformations, and using various loading strategies.

• Profile data
• Choose column data types
• Resolve duplicates and nulls
• Transform data with filters and aggregations
• Transform data with joins and set operators
• Transform data with denormalization and pivots
• Load data with merge, insert, and append
• Exercise - Cleanse, Transform, and Load Data into Unity Catalog
• Module assessment

Implement and manage data quality constraints with Azure Databricks

Learn how to implement and manage data quality constraints in Azure Databricks using Unity Catalog, including validation checks, schema enforcement, and pipeline expectations.

• Implement validation checks
• Implement data type checks
• Detect and manage schema drift
• Manage data quality with pipeline expectations
• Exercise - Implement and Manage Data Quality Constraints with Azure Databricks
• Module assessment

Design and implement data pipelines with Azure Databricks

Learn how to design and implement robust data pipelines using Lakehouse architecture principles, medallion architecture, and Lakeflow Spark Declarative Pipelines in Azure Databricks.

• Design order of operations for a pipeline
• Choose notebook vs Lakeflow Pipelines
• Design Lakeflow job logic
• Design error handling in pipelines and jobs
• Create pipeline with notebook
• Create pipeline with Lakeflow Spark Declarative Pipelines
• Exercise - Design and Implement Data Pipelines with Azure Databricks
• Module assessment

Implement Lakeflow Jobs with Azure Databricks

Learn how to create, configure, schedule, and monitor Lakeflow Jobs in Azure Databricks to automate your data pipelines.

• Create job setup and configuration
• Configure job triggers
• Schedule a job
• Configure job alerts
• Configure automatic restarts
• Exercise - Implement Lakeflow Jobs with Azure Databricks
• Module assessment

Implement development lifecycle processes in Azure Databricks

Learn how to implement development lifecycle processes in Azure Databricks, including Git version control, branching strategies, testing approaches, and Declarative Automation Bundle deployment using the CLI.

• Apply Git version control best practices
• Manage branching and pull requests
• Implement testing strategy
• Configure and package Declarative Automation Bundles
• Deploy bundle with Databricks CLI
• Exercise - Implement Development Lifecycle Processes in Azure Databricks
• Module assessment

Monitor, troubleshoot and optimize workloads in Azure Databricks

Learn how to monitor cluster consumption, troubleshoot job failures, diagnose Spark performance issues, and implement log streaming to Azure Log Analytics for centralized monitoring of Azure Databricks workloads.

• Monitor and manage cluster consumption
• Troubleshoot and repair Lakeflow Jobs
• Troubleshoot Spark jobs and notebooks
• Investigate caching, skewing, spilling, shuffle
• Implement log streaming with Azure Log Analytics
• Exercise - Monitor, Troubleshoot and Optimize Workloads in Azure Databricks
• Module assessment