forked from diegoecab/oci-deal-accelerator
Diagram generation: ref-arch-driven procedure + spec validator + KB enrichment
The diagram path now follows a documented standard procedure (lookup the closest Oracle Architecture Center reference → confirm components → author absolute_layout → spec validator → render → visually verify) and ships persistent guardrails so layout regressions can't recur. Persistent procedure changes (apply to all users, all sessions): - tools/diagram_spec_validator.py — geometry checks (CONTAINER_TOO_THIN, CONTAINER_PADDING_VIOLATION, LABEL_OVERFLOW_PARENT) run BEFORE either renderer (drawio + PPTX). Catches the subnet-collapse / label-overflow bugs that the post-render drawio validator missed. - tools/oci_diagram_gen.py + tools/oci_pptx_diagram_gen.py — call the spec validator before emitting any output. Adds mysql / mysql_heatwave type aliases. - tools/archcenter_pattern_lookup.py — scores against cached page descriptions (not just the 1-line summary), supports --queries for multi-fragment composition, and applies synonym expansion via kb/architecture-center/synonyms.yaml so "LB HA cross AD" matches "load balancer high availability availability domain". - kb/architecture-center/synonyms.yaml — canonical synonym table (load balancer, autonomous database, data guard, …) used by the lookup scorer. KB enrichment: - tools/archcenter_description_fetcher.py + 121 cached _description.md under kb/diagram/assets/archcenter-refs/<slug>/. Removes the runtime dependency on docs.oracle.com when authoring specs and feeds the pattern-lookup scorer. - 110+ cached .drawio / .svg / .png references for offline reuse, plus the OCI Toolkit v24.2 import (kb/diagram/assets/oci-toolkit-drawio). Documentation: - docs/skill/output-formats.md — new "Standard diagram-generation procedure (MANDATORY)" + geometry rules + the new validator entry. - SKILL.md option 2 — references the mandatory procedure. - README.md — describes the spec validator, archcenter_pattern_lookup and description fetcher, and updates the KB-health table. Tooling that backs the procedure (cumulative across recent sessions): tools/archcenter_case_runner.py, archcenter_batch_driver.py, archcenter_zip_downloader.py, drawio_visual_validator.py, drawio_fidelity_eval.py, harvest_drawio_icon.py, import_oci_library.py, oci_pptx_diagram_gen.py, oci_pptx_render.py, refresh_pptx_icon_index.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,308 @@
|
||||
# Data platform - decentralized data platform
|
||||
|
||||
- Source: https://docs.oracle.com/en/solutions/data-platform-decentralized/index.html
|
||||
- Date: 2025-03
|
||||
- Type: reference-architecture
|
||||
- Services: adw, data-catalog, data-integration, object-storage
|
||||
- Tags: data-platform, autonomous
|
||||
|
||||
## Summary (catalog)
|
||||
|
||||
Decentralized data lakehouse with domain-level ADB-S instances sharing data via Cloud Links or Delta Sharing. Centralized catalog, IaC onboarding per domain. Hub-spoke model with OCI backbone routing.
|
||||
|
||||
## Architecture (fetched from source)
|
||||
|
||||
Data Platform - Decentralized Data Platform
|
||||
|
||||
|
||||
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
-
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
-
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Previous
|
||||
Next
|
||||
JavaScript must be enabled to correctly display this content
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
- Data platform - decentralized data platform
|
||||
|
||||
- Data Platform - Decentralized Data
|
||||
Platform
|
||||
|
||||
|
||||
|
||||
|
||||
Data Platform - Decentralized Data
|
||||
Platform
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Use a data lakehouse to collect and analyze event and streaming data from
|
||||
devices in real time and correlate it with a broad range of enterprise data resources to
|
||||
gain the insights you want.
|
||||
|
||||
|
||||
How best to support and empower your organization’s various teams, such as
|
||||
marketing, finance, or logistics, with the flexibility to work with their
|
||||
domain-specific data while also enabling secure cross-domain data sharing and
|
||||
consumption without duplicating data and creating data silos?
|
||||
|
||||
|
||||
Adopt a domain-driven data architecture that provides teams and departments
|
||||
across the organization with the agility and flexibility needed to efficiently use their
|
||||
data and develop the data products essential for their business.
|
||||
|
||||
|
||||
This reference architecture positions the technology solution within the
|
||||
overall business context, where strategic intents drive the creation of measurable
|
||||
strategic outcomes. These outcomes generate new strategic intents, effectively
|
||||
delivering continuous, data-driven business improvements.
|
||||
|
||||
|
||||
Description of the illustration decentralized-business-context.png
|
||||
|
||||
Each domain independently follows the high-level process shown above to
|
||||
create its domain data products. Domain-driven data architectures provide the
|
||||
flexibility that organizations require by avoiding reliance on a single point of
|
||||
contention, such as a fully centralized data platform and IT team, and by fostering
|
||||
agile innovation to produce trusted data products within each domain.
|
||||
|
||||
|
||||
Description of the illustration decentralized-data-platform-overview.png
|
||||
|
||||
decentralized-data-platform-overview-oracle.zip
|
||||
|
||||
|
||||
The objective of each domain is to acquire domain-related data and then to
|
||||
produce data products that are consumed by other domains or final data consumers.
|
||||
|
||||
|
||||
The domains can be:
|
||||
|
||||
|
||||
|
||||
- Source-aligned: Sources data directly from relevant domain data
|
||||
sources, such as enterprise applications, and produces data products that are
|
||||
consumed by aggregate or consumer-aligned domains. These data products represent the
|
||||
source of truth for a particular domain. The data is granular, curated, and
|
||||
foundational within and across domains.
|
||||
|
||||
- Aggregate: Consumes and combines source-aligned data, creating
|
||||
aggregated and added-value data products that foster reuse, reduce duplication, and
|
||||
comprise foundational business logic needed by consumer-aligned domains.
|
||||
|
||||
- Consumer-aligned: Consumes data from source-aligned and aggregate
|
||||
domains to create data products that serve specific use cases and address data
|
||||
consumer's needs within a given domain.
|
||||
|
||||
|
||||
The data domain teams and their subject matter experts (SMEs) have the
|
||||
flexibility to choose the technology needed to curate their data products, reducing the
|
||||
friction and complexity of long technology selection processes, and reducing the time to
|
||||
deliver data products.
|
||||
|
||||
|
||||
The chosen technology is usually determined at an enterprise level so that
|
||||
it adheres to security, scalability, resilience, and high-availability requirements.
|
||||
This architecture assumes that any Oracle Cloud
|
||||
Infrastructure (OCI) service used with a data lakehouse can be leveraged by any domain.
|
||||
|
||||
|
||||
|
||||
Data domain teams often use automation to deploy domain archetypes, making
|
||||
preconfigured technologies available to quickly onboard new domains while ensuring that
|
||||
enterprise-level requirements, such as security, are enforced.
|
||||
|
||||
|
||||
After they are created, data products are then served to other domains or end
|
||||
users and applications. Data products are continuously curated to provide information
|
||||
and insights.
|
||||
|
||||
|
||||
Data products can be of several types. A single data product can be served by
|
||||
using more than one interface.
|
||||
|
||||
|
||||
- Data sets
|
||||
|
||||
- APIs
|
||||
|
||||
- Dashboards
|
||||
|
||||
- Streams
|
||||
|
||||
- AI and machine learning (ML) models that address a specific need
|
||||
|
||||
|
||||
|
||||
|
||||
This reference architecture uses primarily data sharing as the underlying
|
||||
mechanism to provide and consume data products between domains.
|
||||
|
||||
|
||||
Oracle Autonomous Data Warehouse enables data sharing and allows live sharing of data between Autonomous Data
|
||||
Warehouse instances or with versioned data from any technology that is compliant with the Delta
|
||||
Sharing open protocol.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Functional Architecture
|
||||
|
||||
|
||||
|
||||
|
||||
This architecture depicts a decentralized platform where each domain is a
|
||||
subset of the overall data platform and where each domain can choose the technologies and
|
||||
services used.
|
||||
|
||||
|
||||
The architecture uses a data lakehouse to store and provide data,
|
||||
regardless of its shape or form. For simplicity's sake, the architecture will depict a
|
||||
few domains that use a subset of the available data lakehouse services.
|
||||
|
||||
|
||||
A decentralized data platform that uses a data lakehouse architecture
|
||||
provides:
|
||||
|
||||
|
||||
|
||||
- An interoperable and modular lakehouse architecture where data domains
|
||||
can ingest and curate any type of data for any use case
|
||||
|
||||
- Flexibility for each data domain to use the Oracle Cloud
|
||||
Infrastructure (OCI) services needed to support the creation of their data products
|
||||
|
||||
|
||||
- Curation of data products that can be shared securely by using data
|
||||
sharing, streaming, APIs, dashboards, or applications
|
||||
|
||||
- Agility in creating data products, reducing interdomain dependencies
|
||||
except those required for exchange of data products
|
||||
|
||||
- Increased data domain isolation and reduced data interchange complexity
|
||||
by using accepted data interchange mechanisms and contracts to exchange data between
|
||||
domains
|
||||
|
||||
- Increased data governance and data trust because knowledgeable subject
|
||||
matter experts (SMEs) curate data and data products for their domains
|
||||
|
||||
- Ease of onboarding new data domains using infrastructure as code (IaC)
|
||||
to automate deployment using prebuilt and tested Terraform stacks
|
||||
|
||||
- Resource and cost efficiency as data domain teams right-size the
|
||||
specific services they use to create data products
|
||||
|
||||
- Appropriate cost accountability for each data domain with the option of
|
||||
fine-grained cost control within the specific domains
|
||||
|
||||
|
||||
The following diagram illustrates the functional architecture. For
|
||||
simplicity's sake, only four data domains are shown and only some of the data lakehouse
|
||||
capabilities that can be used by data domains are shown.
|
||||
|
||||
|
||||
Description of the illustration decentralized-data-platform-logical.png
|
||||
|
||||
decentralized-data-platform-logical-oracle.zip
|
||||
|
||||
|
||||
Because the particular industry and organization that deploys a decentralized
|
||||
data platform determines the data domains, this reference architecture doesn't prescribe
|
||||
how data domains should be defined. The data domains depicted are just one example.
|
||||
|
||||
|
||||
The architecture focuses on the following logical divisions used by all
|
||||
domains:
|
||||
|
||||
|
||||
|
||||
- Connect, Ingest, Transform
|
||||
Connects to data sources and
|
||||
ingests and refines their data for use in each of the data layers in the
|
||||
architecture.
|
||||
|
||||
|
||||
Source-aligned data domains source data from
|
||||
internal and external data sources and from other domains consuming their data
|
||||
products. Aggregate and Consumer-aligned data domains usually source their data
|
||||
from other domains data products. All domains can source relevant domain data
|
||||
from external sources.
|
||||
|
||||
|
||||
|
||||
- Persist, Curate, Create
|
||||
Facilitates access and
|
||||
navigation of the data to show the current business view. For relational
|
||||
technologies, data may be logically or physically structured in simple
|
||||
relational, longitudinal, dimensional or OLAP forms. For non-relational data,
|
||||
this layer contains one or more pools of data, either output from an analytical
|
||||
process or data optimized for a specific analytical task.
|
||||
|
||||
|
||||
In
|
||||
this layer, ea
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 566 KiB |
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 496 KiB |
@@ -0,0 +1,7 @@
|
||||
The diagram you downloaded is available in these formats:
|
||||
- DRAWIO
|
||||
- SVG
|
||||
|
||||
You can customize them for your organization using the associated tools:
|
||||
- For DRAWIO format, use draw.io for Confluence, online at diagrams.net, or the desktop app. Go to diagrams.net for more information.
|
||||
- For SVG format, use an SVG editor such as Inkscape or Sketsa SVG Editor, which are free and available for Windows, macOS, Linux.
|
||||
Reference in New Issue
Block a user