Data Engineering with Microsoft Fabric

Introduction into Microsoft Fabric

The chapter introduces the data lake approach. It also provides a high-level overview of the building blocks of Microsoft Fabric and how to get started. The Data Mesh architecture is discussed and compared with Microsoft Fabric.

• What is Microsoft Fabric?
• From traditional Data Warehouse to Data Lake
• Data Mesh Architecture
• Working with Task Flow
• Microsoft Fabric Licensing
• Monitor Microsoft Fabric
• LAB: Getting started with Microsoft Fabric

Introduction to Data Lakes

Microsoft Fabric is built on the idea of replacing a traditional data warehouse with a data lake. This module explains why and how the relational data warehouse could be replaced by a file-based data lake.

• Introducing Data Lakes
• Working with Azure Storage
• Storing Data in a Data Lake
• The Medallion Architecture
• Storage Formats in Data Lakes: CSV, Parquet
• Delta Lake
• Other Open Table Formats

Microsoft OneLake

Microsoft OneLake is the OneDrive equivalent for business data: A place to host files (data lake or delta lake) and tables.

• What is OneLake?
• Creating Workspaces
• Workspace Security
• Working with Domains
• Workspaces and Source Control: Azure DevOps and Github integration
• Deployment Pipelines

Storing Data in OneLake

OneLake provides a single, unified, logical data lake for your whole organization. Like OneDrive, OneLake comes automatically with every Microsoft Fabric tenant and is designed to be the single place for all your analytics data.

• Creating a LakeHouse
• Manually loading data in Lakehouse
• The Lakehouse SQL Analytics Endpoint
• Create a semantic model
• Working with Shortcuts
• Shortcuts and Security
• Connecting External Applications with Microsoft OneLake
• Lakehouse Security
• LAB: Setting up Lakehouses in OneLake

Using Copilot in Microsoft Fabric

Copilots can improve the productivity of Fabric Developers, as well as assisting end users in computing and retrieving relevant data. This module explains how Copilot can be used in Microsoft Fabric.

• How Copilot fits into Microsoft Fabric
• Licensing and prerequisites
• Data privacy and role-based security

Getting started with Data Factory

Data Factory allows you to ingest, prepare and transform data from a rich set of data sources like databases, files, cloud data sources,... This chapter illustrates how to use Activities to build pipelines that ingest data in a Lakehouse.

• What is Data Factory?
• Creating Data Pipelines
• The Copy Data Activity
• Working with Copy Job
• Executing and Monitoring Data Pipelines
• LAB: Ingesting data using Pipelines

Building Dynamic Pipelines

Microsoft Fabric Pipelines are used to ingest data into Fabric. By using expressions, variables and parameters, you learn how to make dynamic pipelines.

• Working with Expressions
• Reusing activity output
• Variables and Parameters
• Using Looping and Conditional Logic in pipelines
• Debugging a pipeline
• LAB: Authoring and debugging advanced Pipelines

Ingest and Transform data using Dataflow Gen2

With Dataflows you can visually design data transformations without the need to learn yet another tool or language. Dataflows in Microsoft Fabric are based on Power Query Online.

• What is DataFlow Gen 2
• Creating Queries to load data
• Understanding the Power Query UI
• Applying Transformations
• Query Folding
• Control the Table Destination
• Using Dataflows inside a Pipeline
• Managing connections
• Controlling the Staging
• LAB: Ingesting and Transforming Data using Dataflows

Data Engineering with Spark

Data engineering is the process of designing and building systems that let people collect and analyze raw data from multiple sources and formats. Using popular languages such as Python, SQL and R data can be loaded, transformed and analyzed via interactive notebooks.

• Introducing Apache Spark
• Creating Environments for Apache Spark clusters
• Working with Notebooks in Fabric
• Magic commands
• Spark DataFrames
• Scheduling Notebooks
• Microsoft Fabric decision guide: Copy activity, Dataflow or Spark
• Using Python Notebooks
• LAB: Getting started with Notebooks in Microsoft Fabric

Data wrangling using PySpark and Spark SQL

PySpark and Spark SQL allow users to perform complex data processing tasks with few lines of code using Notebooks.

• Data Cleansing using PySpark
• Grouping and aggregating data in PySpark
• Joining DataFrames
• Using Spark SQL to select and manipulate data
• Visualizing data using Notebooks and DataFrames
• LAB: Data wrangling using PySpark and Spark SQL

Working with Delta Tables

Delta Lake is an optimized storage layer that provides the foundation for storing data and tables in a Fabric lakehouse. Learn how to create, query and optimize Delta Tables in a Microsoft Fabric.

• What is a Delta Lake?
• Working with Delta Tables
• Managing Schema change
• Version and Optimize Delta Tables
• LAB: Working with Delta Tables

Building a Fabric Data Warehouse

A Synapse Data Warehouse is a database that stores data in OneLake and provides a medium to interact with the database using SQL commands.

• The SQL analytics endpoint of the Lakehouse
• Creating tables in a Synapse Data Warehouse
• Ingesting data using pipelines
• Ingesting data using T-SQL
• Querying the Warehouse
• The Default Power BI semantic model
• LAB: Creating and using a Warehouse

Fabric SQL Databases

Sometimes the restrictions on a Fabric Data Warehouse make it difficult to use for applications that are closer to the operational side. With Fabric SQL Databases, an operational database becomes available, with constraints, indexes, and many more features that SQL Server users might be used to.

• What is Fabric SQL Database
• Connecting clients to the database
• Controlling security
• Disaster recovery
• Fabric SQL Database versus Fabric Warehouse

Reporting in Fabric

Power BI transforms your company's data into rich visuals for you to monitor your business and get answers quickly. Learn how to connect to your data stored in Microsoft Fabric using Power BI.

• Creating Power BI Reports
• DirectQuery vs Import with Microsoft OneLake
• Using and configuring Direct Lake Mode
• LAB: Creating Power BI Reports