Edit

Microsoft OneLake patterns and foundational capabilities

This article introduces common OneLake patterns and the platform capabilities you can use to implement them. Use the information in this article to think through how you want to organize your data environment, then choose the patterns that fit your business, technical, and governance needs.

Each pattern describes how to organize data and ownership to accomplish a specific architectural goal. To implement a pattern, you combine one or more foundational OneLake capabilities - data virtualization, open data interoperability, centralized governance, and integrated analytics and AI. Each capability in turn relies on specific product features like shortcuts, mirroring, OneLake security, and Direct Lake mode. The same capability and feature often appear in more than one pattern.

Note

This article is based on patterns identified in the OneLake architectural guidance white paper.

Consider these five patterns as building blocks for your OneLake design. Most environments combine more than one. Choose the patterns that match your goals:

Unified data access with minimal replication

If your data is spread across multiple clouds, on-premises systems, or external lakes, copying it all into one place might not be practical - or even possible. The unified data access with minimal replication pattern treats OneLake as a single logical data layer across those sources. Instead of building ingest pipelines for every source, you use shortcuts to reference data in place and mirroring when you need a synchronized, query-optimized copy.

Use this pattern when:

  • Your data is spread across multiple clouds, on-premises systems, or external lakes.
  • Replicating data into a central store would create excessive storage, latency, or compliance overhead.
  • You need to onboard new sources quickly without authoring full extract, transform, load (ETL) pipelines.
  • You want to preserve investments in existing data lakes, warehouses, and operational stores.

Apply unified data access

To put this pattern into practice, start with two primary data access approaches that don't require you to build or operate data movement processes: virtualization makes source data available through OneLake without copying it, and zero-ETL mirroring brings a platform-managed, synchronized copy into OneLake as analytics-ready Delta tables. Only use Fabric data movement tools when these approaches don't support the source or meet your requirements. For additional guidance on choosing and combining these approaches, see Unify data with OneLake shortcuts and mirroring.

  1. Inventory your data sources to determine which ones OneLake can access through virtualization or zero-ETL mirroring: cloud object storage, external catalogs, operational databases, and Dataverse. Flag any remaining sources as needing a data movement approach.

  2. Choose the right data access technique for each supported source. Prefer virtualization when the source supports no-copy access. Use zero-ETL mirroring when the source requires a synchronized, query-optimized copy:

Source data How to access it Data handling
Cloud object storage (Azure Data Lake Storage Gen2, Amazon S3, Google Cloud Storage) and S3-compatible on-premises storage Shortcuts Virtualization: Makes source data available without copying it
Data managed in an external catalog that you want to make available without copying (for example, Azure Databricks Unity Catalog) Metadata mirroring - syncs only catalog metadata (schemas, tables) and accesses the source data through shortcuts Virtualization: Makes source data available without copying it
Operational databases that need a query-optimized copy (Azure SQL Database, Azure Cosmos DB, Snowflake, PostgreSQL, SQL Server 2025, Oracle Database, Google BigQuery) Database mirroring, or open mirroring for supported custom and partner solutions Zero-ETL mirroring: Creates a synchronized Delta copy
Dataverse (Dynamics 365 and Power Platform data) Shortcuts or Link to Microsoft Fabric for zero-copy access Virtualization: Makes source data available without copying it

  1. Transform source data when needed. Shortcut transformations can process supported files exposed through a shortcut, whether the files are stored externally or already in OneLake. Use shortcut file transformations to convert structured files into Delta tables or shortcut AI transformations to process unstructured text. Shortcut transformations create transformed Delta output and keep it synchronized with the data referenced by the shortcut.

  2. Use Fabric data movement tools when virtualization and mirroring don't support a source or when you need complex transformations, orchestration, a scheduled movement cadence, or streaming ingestion. For help choosing among pipelines, dataflows, copy jobs, and eventstreams, see Choose a data movement strategy.

When you do choose data movement, land copied data in an open table format, such as Delta Parquet or Iceberg. Mirroring and shortcut transformations already create Delta output. Using open formats keeps virtualized data, synchronized copies, and transformed Delta output readable by Fabric engines and external platforms.

  1. Record the reason whenever you create a synchronized copy, transformed Delta output, or a copy through Fabric data movement tools. This record keeps the decision auditable. Create a copy only when a source needs a physical, query-optimized layout or can't meet your freshness, transformation-cost, compliance, or processing requirements virtually.

  2. Apply OneLake security to the data made available through OneLake so the same policies cover virtualized data, synchronized copies, and transformed Delta output.

  3. Endorse and describe the resulting data items in the OneLake catalog so consumers can find and trust them.

Unified data access capabilities

  • Data virtualization and zero-ETL mirroring - Expose data that lives in other systems and clouds through no-copy references or synchronized, analytics-ready copies. Features:
  • Centralized governance - Apply consistent security and discovery to virtualized sources just as you would to native OneLake data. Features:
  • Open data interoperability - Keep virtualized data and platform-managed copies readable by both Fabric engines and external platforms. Features:

Medallion architecture (bronze, silver, gold)

Making data available in OneLake is only the first step. Raw data from source systems usually isn't safe to use directly for analytics or AI. It often contains duplicates, errors, inconsistent formats, or sensitive fields. When multiple teams build on the same source data, they need a shared definition of what each stage of data is trusted for.

The medallion architecture pattern organizes data in OneLake into three quality layers: bronze for raw, immutable source data; silver for cleansed and conformed data; and gold for certified, business-ready tables and semantic models. Each layer is a defined stage that downstream consumers can depend on. Silver and gold tables are reusable across BI, analytics, and AI workloads, so teams don't rebuild the same cleansing or modeling logic in separate tools.

Use this pattern when:

  • Multiple teams build on the same source data and need consistent quality.
  • You need traceable lineage from raw inputs to certified outputs.
  • You need a clear contract between data engineering and analytics or AI consumers.

For more information about this pattern, see Understand medallion architecture for Fabric with OneLake. That article covers layer design, deployment models, storage formats, materialized lake views, and Delta table optimization.

How to apply it

A working medallion hinges on one idea: each layer is a contract with downstream consumers, and data only advances to the next layer after it meets that layer's quality standards.

  1. Identify your raw sources and the consumers who depend on certified data.

  2. Define what belongs in each layer, and apply these definitions consistently across domains:

    Layer Contents Typical consumers
    Bronze Raw, immutable data captured directly from sources with no schema enforcement Data engineers (limited access)
    Silver Cleansed, deduplicated, and conformed to shared business definitions Data engineers and trained analysts
    Gold Curated, business-ready tables and semantic models All BI, analytics, and AI consumers

  3. Produce each layer with the right Fabric workload - typically Data Engineering (Spark) or Data Factory for bronze and silver, and Data Warehouse or Power BI semantic models for gold. Preserve source fidelity in bronze by using the original format, a shortcut to source data, Parquet, or Delta as appropriate. Use Delta tables for silver and gold so Fabric workloads can reliably read and write the refined data.

  4. Apply layer-aware access policies. Use OneLake security for supported items and the applicable Fabric and SQL permissions for warehouses. Restrict access to bronze, make silver available to analysts, and grant access to gold based on consumer needs and least privilege standards.

  5. Use curated gold outputs for downstream analytics. Build gold-layer semantic models on Direct Lake mode so Power BI can read OneLake data without creating an imported copy or requiring scheduled refreshes.

  6. Confirm that every gold output has traceable lineage through silver to its bronze sources. Then, endorse gold-layer tables and semantic models as certified in the OneLake catalog. This validation helps consumers identify which data is ready for production use.

  7. Reuse gold semantic models to jump-start Fabric IQ ontologies. This step gives AI agents governed business context grounded in certified data.

Foundational capabilities

  • Integrated analytics and AI - Bronze, silver, and gold layers feed every analytics and AI workload on OneLake without engine-specific copies. Features:
  • Centralized governance - Apply different access policies and quality gates at each layer so consumers only see data appropriate to their role. Features:
  • Open data interoperability - Store the layers in open formats so external engines can read them alongside Fabric. Features:

Domain-oriented data mesh on a shared platform

If you have multiple business teams producing and consuming data, routing every request through a single central data team can slow delivery. Business teams often understand their own data and requirements best, but decentralizing ownership without shared governance can lead to inconsistent security, quality, and lineage.

The domain-oriented data mesh pattern gives each business domain ownership of its own data products while all domains follow shared standards on a OneLake foundation. Each domain publishes its own data products, and other domains access them through shortcuts and consume them with Fabric analytics and AI workloads. Centralized identity, security, and governance policies apply uniformly across all domains.

Use this pattern when:

  • A single central data team becomes a bottleneck for delivery.
  • Different business domains have distinct data, requirements, and release cadences.
  • You need clear accountability for data quality at the domain level without giving up enterprise-wide governance.

Apply a domain-oriented data mesh

Find the right balance between decentralization and consistency. Push ownership to the domain that knows the data best, and keep identity, security, and lineage centralized so every domain's data products meet the same standards.

  1. Identify your business domains. Each domain should represent a coherent area of the business with a team that can own and operate its data products end-to-end.

  2. Create a domain for each business area and assign workspaces to it. Set up a separate central domain for shared infrastructure and reusable enterprise data.

  3. Define data product standards that every domain must meet - for example, endorsement or certification requirements, documented schemas, ownership metadata, versioning, and service-level agreements (SLAs). These standards make each product a reusable, discoverable contract rather than just a workspace folder.

  4. Use OneLake security to apply role-based data access controls at the folder, table, row, and column level so producers can publish data products without exposing everything in their workspace.

  5. Apply tenant-wide governance with the OneLake catalog for cross-domain discovery and lineage, and Microsoft Purview for sensitivity labels and audit. Extend the same identity and policy model to AI agents that consume domain data products, so agent access is governed like any other consumer's.

  6. Have consumer domains use shortcuts to reference producer data products rather than copying them. Consumers can then use the referenced data products in the Fabric workload that fits their needs. For Power BI semantic models, use Direct Lake mode to read data directly from OneLake. Use Fabric Data Agents or Fabric IQ to create AI experiences grounded in governed domain data products.

  7. If domains publish to catalogs outside Fabric, plan for access-control synchronization so permissions stay consistent between OneLake and the external catalog.

    Tip

    The Microsoft open-source accelerator Policy Weaver can automate this synchronization for Azure Databricks (Unity Catalog), Snowflake, and Dataverse sources. It mirrors data-access policies into OneLake security roles, complementing mirroring (which moves data but not permissions).

Data mesh capabilities

  • Centralized governance - Decentralize ownership to domains while keeping identity, security, and lineage centralized. Features:
  • Data virtualization - Let consumer domains use producer-owned data products through references rather than copies. Features:
    • Shortcuts enable zero-copy sharing between domains.
  • Integrated analytics and AI - Make every domain's data products consumable across Fabric workloads. Features:

Platform consolidation for analytics and AI

If you run several analytics platforms side by side - separate tools for data warehousing, business intelligence, data science, real-time analytics, and AI - each tool comes with its own data copies, pipelines, and governance model. That fragmentation drives up cost and makes it hard to apply consistent security or get a single answer to a business question.

The platform consolidation pattern brings these workloads onto Fabric, where OneLake provides a shared, governed data foundation. Fabric workloads access, transform, synchronize, or analyze data through this foundation instead of relying on separate data and governance models for each tool.

Use this pattern when:

  • You're using multiple analytics platforms with overlapping capabilities.
  • Engine-specific data copies and pipelines drive cost and maintenance overhead.
  • You need a single governance and security model across all analytics and AI workloads.

Apply platform consolidation

Aim for fewer platforms, not more integrations. Consolidate workloads in Fabric instead of bridging tools together, and bridge external engines only when you can't retire them yet.

  1. Inventory the analytics, data warehousing, data science, business intelligence (BI), and AI tools and pipelines you use today. Note which workloads each tool serves and what data it copies.

  2. Map each existing workload to the Fabric workload that can replace it:

    Legacy workload Fabric workload
    Data orchestration and ETL Data Factory
    Spark notebooks and lakehouse processing Data Engineering
    SQL data warehousing Data Warehouse
    Streaming and KQL analytics Real-Time Intelligence
    ML model training and experiment tracking Data Science
    Operational databases Databases (SQL database in Fabric and Cosmos DB in Fabric)
    BI visualization and semantic models Power BI with Direct Lake mode
    Conversational AI grounded on enterprise data Fabric Data Agents, Copilot for Fabric, Fabric IQ

  3. Establish one governance and security model across all workloads using OneLake security, Microsoft Purview, and the OneLake catalog. Configure customer-managed keys when supported Fabric items require another layer of encryption.

  4. Consolidate analytical data in OneLake by using Delta or Iceberg format so workloads can share a governed data foundation. Include operational workloads by consolidating them onto Fabric Databases, which make synchronized analytical data available in OneLake.

  5. Ground AI on the consolidated data. Build ontologies (preview) over your curated data layer and expose them to agents through the Ontology MCP server, so Fabric Data Agents, Microsoft 365 Copilot, and external tools reason over the same governed context. You can generate ontology definitions from Power BI semantic models in Import, Direct Lake, or DirectQuery mode. Use Direct Lake mode when you need generated bindings to supported OneLake data, and review the current ontology limitations.

  6. For external engines you can't retire yet, expose OneLake data to them through Azure Databricks integration, Iceberg interoperability with Snowflake, or OneLake access and APIs.

  7. Retire the replaced tools, data copies, and pipelines after you validate the Fabric equivalent. That way the consolidation removes cost, licenses, and hand-offs rather than adding another platform to the pile.

Platform consolidation capabilities

External data sharing across organizations

If you exchange data with partners, suppliers, customers, or other divisions on an ongoing basis, batch exports, file transfers, and duplicated downstream systems add latency, cost, and governance gaps. The external data sharing pattern gives consumers outside your organization or business division direct access to curated OneLake data without recurring exports. Consumers can access the data through Fabric cross-tenant sharing or from external analytics platforms such as Snowflake and Azure Databricks by using OneLake interoperability capabilities.

Consumers see updates as you publish them. You control access to the source data through the sharing or interoperability mechanism that supports the consumer's platform.

Use this pattern when:

  • You exchange data with external organizations on an ongoing basis.
  • Batch exports or file transfers add latency, complexity, or governance gaps.
  • You need to track and revoke external access centrally.

Apply external data sharing

External sharing works best when you use virtualization instead of exporting data. Match the access method to what each consumer can read, and apply the access controls supported by that sharing or interoperability mechanism.

  1. Identify the data products you want to share externally and the consumers who need them (partners, suppliers, customers). Typically, you share curated tables and files that are well-defined and documented.

  2. Choose the right sharing approach for each consumer:

    Consumer type Recommended approach
    Fabric users in another tenant External data sharing for read-only, virtualized cross-tenant access
    Snowflake on Azure users Iceberg interoperability with Snowflake to read Fabric tables exposed in Iceberg format
    Azure Databricks users OneLake catalog federation in Azure Databricks to query OneLake tables through Unity Catalog without copying data
    Applications or tools that support ADLS Gen2 or Blob APIs OneLake access and APIs to access OneLake data through supported APIs

    To bring Dataverse data into OneLake before sharing it, use the unified data access pattern.

  3. Scope external access with the permissions supported by the selected sharing mechanism. For Fabric external data sharing, the share grants read-only access to any user in the invited user's home tenant. Provider-side security and governance policies, including OneLake security, sensitivity labels, and data loss prevention policies, aren't enforced in the consumer's tenant. The consumer must govern downstream access in their environment.

  4. Agree on the terms of each sharing relationship up front - what is shared, with whom, and for how long. For Fabric external data sharing, revoke access from the External data shares tab on the Manage permissions page. For other approaches, revoke access through the selected sharing mechanism. Confirm that the consumer loses visibility.

  5. Apply sensitivity labels, audit, and data loss prevention with Microsoft Purview in the provider's Fabric environment.

  6. Endorse and document the source data products in the OneLake catalog so providers can find and govern them before sharing. The OneLake catalog doesn't publish data products to external tenants or analytics platforms.

External data sharing capabilities

  • Data virtualization - Share data through no-copy references without managing export pipelines. Features:
  • Open data interoperability - Share with consumers who don't use Fabric by publishing in open formats. Features:
  • Centralized governance - Govern source data in Fabric and control external access through each sharing mechanism. Features:
    • OneLake security scopes access to source data in Fabric.
    • Microsoft Purview applies sensitivity labels, audit, and data loss prevention in the provider's Fabric environment.
    • OneLake catalog supports provider-side discovery and endorsement before sharing.
  • Last updated on