diegoecab/oci-deal-accelerator
1
1
Fork 2
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4552ef7226acb2092eb79b7861bcd3b6a7630ee3
oci-deal-accelerator/kb/diagram/assets/archcenter-refs/oci-azure-multicloud-data-solution/_description.md
root b30a4f0d32 Diagram generation: ref-arch-driven procedure + spec validator + KB enrichment 
The diagram path now follows a documented standard procedure (lookup
the closest Oracle Architecture Center reference → confirm components
→ author absolute_layout → spec validator → render → visually verify)
and ships persistent guardrails so layout regressions can't recur.

Persistent procedure changes (apply to all users, all sessions):
- tools/diagram_spec_validator.py — geometry checks (CONTAINER_TOO_THIN,
  CONTAINER_PADDING_VIOLATION, LABEL_OVERFLOW_PARENT) run BEFORE either
  renderer (drawio + PPTX). Catches the subnet-collapse / label-overflow
  bugs that the post-render drawio validator missed.
- tools/oci_diagram_gen.py + tools/oci_pptx_diagram_gen.py — call the
  spec validator before emitting any output. Adds mysql / mysql_heatwave
  type aliases.
- tools/archcenter_pattern_lookup.py — scores against cached page
  descriptions (not just the 1-line summary), supports --queries for
  multi-fragment composition, and applies synonym expansion via
  kb/architecture-center/synonyms.yaml so "LB HA cross AD" matches
  "load balancer high availability availability domain".
- kb/architecture-center/synonyms.yaml — canonical synonym table
  (load balancer, autonomous database, data guard, …) used by the
  lookup scorer.

KB enrichment:
- tools/archcenter_description_fetcher.py + 121 cached _description.md
  under kb/diagram/assets/archcenter-refs/<slug>/. Removes the runtime
  dependency on docs.oracle.com when authoring specs and feeds the
  pattern-lookup scorer.
- 110+ cached .drawio / .svg / .png references for offline reuse,
  plus the OCI Toolkit v24.2 import (kb/diagram/assets/oci-toolkit-drawio).

Documentation:
- docs/skill/output-formats.md — new "Standard diagram-generation
  procedure (MANDATORY)" + geometry rules + the new validator entry.
- SKILL.md option 2 — references the mandatory procedure.
- README.md — describes the spec validator, archcenter_pattern_lookup
  and description fetcher, and updates the KB-health table.

Tooling that backs the procedure (cumulative across recent sessions):
tools/archcenter_case_runner.py, archcenter_batch_driver.py,
archcenter_zip_downloader.py, drawio_visual_validator.py,
drawio_fidelity_eval.py, harvest_drawio_icon.py, import_oci_library.py,
oci_pptx_diagram_gen.py, oci_pptx_render.py, refresh_pptx_icon_index.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 21:15:21 -03:00

8.4 KiB
Raw Blame History

Establish a multicloud data solution between OCI and Microsoft Azure

Summary (catalog)

Cross-cloud data integration between OCI and Azure. OCI Data Integration for ETL between Azure Blob Storage and OCI Object Storage. ADB-S for analytics with data from both clouds.

Architecture (fetched from source)

Architecture

This reference architecture illustrates an enterprise multicloud data pipeline that collects and formats data from various sources, transferring it to the enterprise data lake or data warehouse. It includes batch integration, data integration, and real-time integration scenarios.

Oracle Interconnect for Microsoft Azure links Azure ExpressRoute and Oracle Cloud Infrastructure FastConnect to connect two separate cloud networks efficiently.

Azure 's Virtual Network (VNet) traffic routes through a private interconnection to OCI's virtual cloud network (VCN).

The following diagram illustrates this reference architecture.

Description of the illustration oci-azure-multicloud-data-solution-diagram.png

oci-azure-multicloud-data-solution-diagram-oracle.zip

OCI Data Integration connects, and extracts data from, on-premises and cloud sources using native adapters, accesses Oracle SaaS applications using BICC connector, conducts transformations on the data, and loads it into an OCI data lake through adapters ( Oracle Autonomous Database or OCI Object Storage ).

Oracle application integration services collect real-time data from diverse source systems such as Oracle SaaS applications, internet-of-things (IoT), streaming services, social media, on-premises systems, and other cloud providers via native adapters. It then executes transformation and orchestration processes before loading the data into an OCI data lake using adapters ( Oracle Autonomous Database or OCI Object Storage ).

OCI GoldenGate captures data from Oracle Autonomous Database and replicates it to Azure Data Lake Gen2 and Azure Synapse Analytics in near real-time via OCI FastConnect . The replication to Synapse involves staging and merging the change data in micro-batches in Azure Data Lake Storage Gen2 before merging it into the Synapse target table.

Flow of events

  • Data extraction and transfer

  • Customer data is transferred from the data source to OCI Object Storage either directly or via default, source-specific drivers.

  • On-premises flat files are moved to OCI Object Storage using the customer's Python script or by establishing an FTP connection with OCI Object Storage for seamless connectivity to Oracle Integration Cloud Service .

  • Data is securely uploaded in its raw form to OCI Object Storage buckets with encryption.

  • Data ingestion and transformation

  • OCI Data Integration retrieves data from OCI Object Storage and other sources, transforms it according to business needs using Apache Spark and a proposed architecture flow, then stores the transformed data back into OCI Object Storage alongside the autonomous database.

  • This process follows the Delta Lake architecture for active ACID properties and compression. The data is now structured, can be queried, and is ready for further analytics.

  • OCI Logging manages all processing logs.

  • Orchestration and scheduling

  • OCI Data Integration manages data flow processes, scheduling the execution of Data Flow applications and Data Science notebooks as necessary.

  • Developers can run Data Flow applications from the UI or Data Science service notebooks for flexibility.

  • Data archival

  • OCI Object Storage lifecycle policies, which are defined and implemented by customers, play a crucial role in automating the process of data archival. These policies facilitate the seamless shifting of data to more cost-effective storage tiers or the systematic deletion of outdated information, all in accordance with predefined rules and guidelines. This automation is essential for ensuring not only efficient data management but also compliance with various retention policies that organizations must adhere to.

  • By utilizing these lifecycle policies, customers can optimize their storage costs while maintaining control over their data retention practices and ensuring that they are aligned with legal and regulatory requirements.

  • Data replication to Azure

  • OCI GoldenGate is used for data replication to Azure via a dedicated network established with Oracle Interconnect for Microsoft Azure .

  • OCI GoldenGate integrates closely with Azure Data Lake and Azure Synapse Analytics for seamless data loading.

  • Data analysis and reporting

  • Oracle Analytics Cloud and Power BI are examples of business intelligence tools that can establish a connection with OCI Object Storage or Oracle Autonomous Database .

  • These tools gather the data that has been transformed and produce user-friendly dashboards showcasing key business key performance indicators (KPIs).

  • Through these dashboards, valuable insights can be obtained from the data, facilitating well-informed decision-making.

The architecture has the following components:

  • Tenancy A tenancy is a secure and isolated partition that Oracle sets up within Oracle Cloud when you sign up for Oracle Cloud Infrastructure . You can create, organize, and administer your resources in Oracle Cloud within your tenancy. A tenancy is synonymous with a company or organization. Usually, a company will have a single tenancy and reflect its organizational structure within that tenancy. A single tenancy is usually associated with a single subscription, and a single subscription usually only has one tenancy.

  • Region An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).

  • Compartment Compartments are cross-region logical partitions within an Oracle Cloud Infrastructure tenancy. Use compartments to organize your resources in Oracle Cloud , control access to the resources, and set usage quotas. To control access to the resources in a given compartment, you define policies that specify who can access the resources and what actions they can perform.

  • Availability domains Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain shouldn't affect the other availability domains in the region.

  • Virtual cloud network (VCN) and subnets A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.

  • ExpressRoute Azure ExpressRoute lets you set up a private connection between a VNet and another network, such as your on-premises network or a network in another cloud provider.

Azure ExpressRoute is a more reliable and faster alternative to typical internet connections because the traffic over Azure ExpressRoute does not traverse the public internet.

  • Autonomous Database

Oracle Autonomous Database is a fully managed, preconfigured database environments that you can use for transaction processing and data warehousing workloads. You do not need to configure or manage any hardware, or install any software. Oracle Cloud Infrastructure handles creating the database, as well as backing up, patching, upgrading

Reference in New Issue View Git Blame Copy Permalink