b30a4f0d32
The diagram path now follows a documented standard procedure (lookup the closest Oracle Architecture Center reference → confirm components → author absolute_layout → spec validator → render → visually verify) and ships persistent guardrails so layout regressions can't recur. Persistent procedure changes (apply to all users, all sessions): - tools/diagram_spec_validator.py — geometry checks (CONTAINER_TOO_THIN, CONTAINER_PADDING_VIOLATION, LABEL_OVERFLOW_PARENT) run BEFORE either renderer (drawio + PPTX). Catches the subnet-collapse / label-overflow bugs that the post-render drawio validator missed. - tools/oci_diagram_gen.py + tools/oci_pptx_diagram_gen.py — call the spec validator before emitting any output. Adds mysql / mysql_heatwave type aliases. - tools/archcenter_pattern_lookup.py — scores against cached page descriptions (not just the 1-line summary), supports --queries for multi-fragment composition, and applies synonym expansion via kb/architecture-center/synonyms.yaml so "LB HA cross AD" matches "load balancer high availability availability domain". - kb/architecture-center/synonyms.yaml — canonical synonym table (load balancer, autonomous database, data guard, …) used by the lookup scorer. KB enrichment: - tools/archcenter_description_fetcher.py + 121 cached _description.md under kb/diagram/assets/archcenter-refs/<slug>/. Removes the runtime dependency on docs.oracle.com when authoring specs and feeds the pattern-lookup scorer. - 110+ cached .drawio / .svg / .png references for offline reuse, plus the OCI Toolkit v24.2 import (kb/diagram/assets/oci-toolkit-drawio). Documentation: - docs/skill/output-formats.md — new "Standard diagram-generation procedure (MANDATORY)" + geometry rules + the new validator entry. - SKILL.md option 2 — references the mandatory procedure. - README.md — describes the spec validator, archcenter_pattern_lookup and description fetcher, and updates the KB-health table. Tooling that backs the procedure (cumulative across recent sessions): tools/archcenter_case_runner.py, archcenter_batch_driver.py, archcenter_zip_downloader.py, drawio_visual_validator.py, drawio_fidelity_eval.py, harvest_drawio_icon.py, import_oci_library.py, oci_pptx_diagram_gen.py, oci_pptx_render.py, refresh_pptx_icon_index.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8.3 KiB
Data platform - decentralized data platform
- Source: https://docs.oracle.com/en/solutions/data-platform-decentralized/index.html
- Date: 2025-03
- Type: reference-architecture
- Services: adw, data-catalog, data-integration, object-storage
- Tags: data-platform, autonomous
Summary (catalog)
Decentralized data lakehouse with domain-level ADB-S instances sharing data via Cloud Links or Delta Sharing. Centralized catalog, IaC onboarding per domain. Hub-spoke model with OCI backbone routing.
Architecture (fetched from source)
Data Platform - Decentralized Data Platform
Previous Next JavaScript must be enabled to correctly display this content
-
Data platform - decentralized data platform
-
Data Platform - Decentralized Data Platform
Data Platform - Decentralized Data Platform
Use a data lakehouse to collect and analyze event and streaming data from devices in real time and correlate it with a broad range of enterprise data resources to gain the insights you want.
How best to support and empower your organization’s various teams, such as marketing, finance, or logistics, with the flexibility to work with their domain-specific data while also enabling secure cross-domain data sharing and consumption without duplicating data and creating data silos?
Adopt a domain-driven data architecture that provides teams and departments across the organization with the agility and flexibility needed to efficiently use their data and develop the data products essential for their business.
This reference architecture positions the technology solution within the overall business context, where strategic intents drive the creation of measurable strategic outcomes. These outcomes generate new strategic intents, effectively delivering continuous, data-driven business improvements.
Description of the illustration decentralized-business-context.png
Each domain independently follows the high-level process shown above to create its domain data products. Domain-driven data architectures provide the flexibility that organizations require by avoiding reliance on a single point of contention, such as a fully centralized data platform and IT team, and by fostering agile innovation to produce trusted data products within each domain.
Description of the illustration decentralized-data-platform-overview.png
decentralized-data-platform-overview-oracle.zip
The objective of each domain is to acquire domain-related data and then to produce data products that are consumed by other domains or final data consumers.
The domains can be:
-
Source-aligned: Sources data directly from relevant domain data sources, such as enterprise applications, and produces data products that are consumed by aggregate or consumer-aligned domains. These data products represent the source of truth for a particular domain. The data is granular, curated, and foundational within and across domains.
-
Aggregate: Consumes and combines source-aligned data, creating aggregated and added-value data products that foster reuse, reduce duplication, and comprise foundational business logic needed by consumer-aligned domains.
-
Consumer-aligned: Consumes data from source-aligned and aggregate domains to create data products that serve specific use cases and address data consumer's needs within a given domain.
The data domain teams and their subject matter experts (SMEs) have the flexibility to choose the technology needed to curate their data products, reducing the friction and complexity of long technology selection processes, and reducing the time to deliver data products.
The chosen technology is usually determined at an enterprise level so that it adheres to security, scalability, resilience, and high-availability requirements. This architecture assumes that any Oracle Cloud Infrastructure (OCI) service used with a data lakehouse can be leveraged by any domain.
Data domain teams often use automation to deploy domain archetypes, making preconfigured technologies available to quickly onboard new domains while ensuring that enterprise-level requirements, such as security, are enforced.
After they are created, data products are then served to other domains or end users and applications. Data products are continuously curated to provide information and insights.
Data products can be of several types. A single data product can be served by using more than one interface.
-
Data sets
-
APIs
-
Dashboards
-
Streams
-
AI and machine learning (ML) models that address a specific need
This reference architecture uses primarily data sharing as the underlying mechanism to provide and consume data products between domains.
Oracle Autonomous Data Warehouse enables data sharing and allows live sharing of data between Autonomous Data Warehouse instances or with versioned data from any technology that is compliant with the Delta Sharing open protocol.
Functional Architecture
This architecture depicts a decentralized platform where each domain is a subset of the overall data platform and where each domain can choose the technologies and services used.
The architecture uses a data lakehouse to store and provide data, regardless of its shape or form. For simplicity's sake, the architecture will depict a few domains that use a subset of the available data lakehouse services.
A decentralized data platform that uses a data lakehouse architecture provides:
-
An interoperable and modular lakehouse architecture where data domains can ingest and curate any type of data for any use case
-
Flexibility for each data domain to use the Oracle Cloud Infrastructure (OCI) services needed to support the creation of their data products
-
Curation of data products that can be shared securely by using data sharing, streaming, APIs, dashboards, or applications
-
Agility in creating data products, reducing interdomain dependencies except those required for exchange of data products
-
Increased data domain isolation and reduced data interchange complexity by using accepted data interchange mechanisms and contracts to exchange data between domains
-
Increased data governance and data trust because knowledgeable subject matter experts (SMEs) curate data and data products for their domains
-
Ease of onboarding new data domains using infrastructure as code (IaC) to automate deployment using prebuilt and tested Terraform stacks
-
Resource and cost efficiency as data domain teams right-size the specific services they use to create data products
-
Appropriate cost accountability for each data domain with the option of fine-grained cost control within the specific domains
The following diagram illustrates the functional architecture. For simplicity's sake, only four data domains are shown and only some of the data lakehouse capabilities that can be used by data domains are shown.
Description of the illustration decentralized-data-platform-logical.png
decentralized-data-platform-logical-oracle.zip
Because the particular industry and organization that deploys a decentralized data platform determines the data domains, this reference architecture doesn't prescribe how data domains should be defined. The data domains depicted are just one example.
The architecture focuses on the following logical divisions used by all domains:
- Connect, Ingest, Transform Connects to data sources and ingests and refines their data for use in each of the data layers in the architecture.
Source-aligned data domains source data from internal and external data sources and from other domains consuming their data products. Aggregate and Consumer-aligned data domains usually source their data from other domains data products. All domains can source relevant domain data from external sources.
- Persist, Curate, Create Facilitates access and navigation of the data to show the current business view. For relational technologies, data may be logically or physically structured in simple relational, longitudinal, dimensional or OLAP forms. For non-relational data, this layer contains one or more pools of data, either output from an analytical process or data optimized for a specific analytical task.
In this layer, ea