Data Engineering with Microsoft Fabric - Advanced Concepts and Patterns

Loading and Cleansing Data

Incremental data ingestion with pipelines

Microsoft Fabric Pipelines support efficient incremental data ingestion. Learn how to detect data changes, process only new or modified data, and build scalable ingestion patterns.

• Incremental ingestion concepts and use cases
• Using watermarks and high-water-mark patterns
• Change detection strategies (timestamps, keys, CDC)
• Implementing incremental logic in Fabric pipelines
• Handling late-arriving and updated data
• LAB: Building an incremental ingestion pipeline

Declarative ETL with Materialized lake views

Materialized Lake Views store precomputed query results in OneLake to improve performance and reuse data across Fabric.

• Materialized Lake Views architecture and storage model
• Authoring Materialized Lake Views with SQL
• Refresh behavior and incremental processing
• Performance and optimization
• Consuming Materialized Lake Views across Fabric workloads
• Security, governance, and lineage considerations

Syncing data using Microsoft Fabric Mirroring

Fabric Mirroring enables near real-time replication of data from operational systems into OneLake, allowing analytics workloads to run directly on continuously updated source data without complex ingestion pipelines.

• What Fabric Mirroring is and when to use it
• Supported source systems and prerequisites
• Setting up a mirrored database in Fabric
• Understanding change data capture (CDC) and latency
• Accessing mirrored data through Lakehouse and Warehouse endpoints
• Security, schema evolution, and operational considerations
• Using mirrored data for analytics and Power BI reporting
• LAB: Creating and querying a mirrored database

Update your data from within Power BI

Fabric User Data Functions enable you to encapsulate reusable business logic directly within Microsoft Fabric, supporting translytical task flows that seamlessly combine analytical insights with operational actions across notebooks, pipelines, and Power BI.

• Overview of User Data Functions and Translytical task flow concepts
• Creating User Data Functions in Microsoft Fabric
• Implementing functions using notebooks and Spark
• Invoking User Data Functions from notebooks and pipelines
• Using User Data Functions in Power BI to drive translytical actions
• Parameter handling, performance, and scalability considerations
• Security, versioning, and governance of shared business logic
• LAB: Building and executing a translytical task flow with User Data Functions

Real-Time Data

Real-Time Analytics in Fabric

Real-Time Analytics is a fully managed big data analytics platform optimized for streaming, time-series data. It contains a dedicated query language and engine with for searching structured, semi-structured, and unstructured data in close to real-time.

• Create an Eventhouse and KQL database
• Ingesting data into tables
• Working with EventStream
• Query data using Kusto Query Language (KQL)
• Building real-time dashboards
• LAB: Working with Real-Time Analytics

Respond to Events with Data Activator

Data Activator in Microsoft Fabric takes action based on what's happening in your data. Learn how to setup conditions against your data and trigger actions like run a Power Automate Flow when the conditions are met.

• Creating and using Reflexes
• Defining Triggers, Conditions and Actions
• Getting data from Reports or Eventstreams
• LAB: Use Data Activator in Fabric

Using DevOps in Fabric

Fabric DevOps: Git Integration

Microsoft Fabric supports seamless Git-based source control through both GitHub and Azure DevOps. Linking a workspace to a repository enables versioning, collaboration, branching strategies, and controlled change management across development teams.

• Basic Concepts in Git Integration
• Managing Branches in Microsoft Fabric Workspaces
• Handling Errors and Conflicts
• Git Integration Source Code Format
• LAB: Git Integration

Fabric DevOps: Deployment Pipelines

Microsoft Fabric offers built-in deployment pipelines that streamline the promotion of content across development, test, and production environments. Variable Libraries further simplify configuration management during deployments.

• Options for Deploying Resources
• Setting Up Microsoft Fabric Deployment Pipelines
• Working with Microsoft Fabric Variable Libraries
• LAB: Setup Deployment Pipelines

AI in Microsoft Fabric

Improve AI with Fabric IQ and Ontologies

Fabric IQ brings AI-powered intelligence into Microsoft Fabric by grounding generative AI experiences in your data. Ontologies provide the semantic layer that helps Fabric IQ understand business concepts, relationships, and context, enabling more accurate insights, queries, and Copilot experiences.

• Overview of Fabric IQ and AI-enabled experiences in Fabric
• Ontologies in Fabric IQ
• Defining business entities, relationships, and metadata
• Connecting ontologies to Lakehouse and Warehouse data
• Using ontologies to improve Copilot queries and insights
• Governance, security, and lifecycle management of ontologies
• Best practices for modeling semantic knowledge in Fabric
• LAB: Creating an ontology and using Fabric IQ to explore data

Fabric Data Agents: Chat with your Data

With a Fabric data agent, your team can have conversations, with plain English-language questions, about the data that your organization stored in Fabric OneLake and then receive relevant answers. This way, even people without technical expertise in AI or a deep understanding of the data structure can receive precise and context-rich answers.

• The Purpose of the Fabric Data Agents
• Creating and Publishing a Fabric Data Agent
• Interacting with a Fabric Data Agent
• Understanding Permission Delegation
• Fine-tuning the Fabric Data Agent with Instructions and Examples