forked from diegoecab/oci-deal-accelerator
Diagram generation: ref-arch-driven procedure + spec validator + KB enrichment
The diagram path now follows a documented standard procedure (lookup the closest Oracle Architecture Center reference → confirm components → author absolute_layout → spec validator → render → visually verify) and ships persistent guardrails so layout regressions can't recur. Persistent procedure changes (apply to all users, all sessions): - tools/diagram_spec_validator.py — geometry checks (CONTAINER_TOO_THIN, CONTAINER_PADDING_VIOLATION, LABEL_OVERFLOW_PARENT) run BEFORE either renderer (drawio + PPTX). Catches the subnet-collapse / label-overflow bugs that the post-render drawio validator missed. - tools/oci_diagram_gen.py + tools/oci_pptx_diagram_gen.py — call the spec validator before emitting any output. Adds mysql / mysql_heatwave type aliases. - tools/archcenter_pattern_lookup.py — scores against cached page descriptions (not just the 1-line summary), supports --queries for multi-fragment composition, and applies synonym expansion via kb/architecture-center/synonyms.yaml so "LB HA cross AD" matches "load balancer high availability availability domain". - kb/architecture-center/synonyms.yaml — canonical synonym table (load balancer, autonomous database, data guard, …) used by the lookup scorer. KB enrichment: - tools/archcenter_description_fetcher.py + 121 cached _description.md under kb/diagram/assets/archcenter-refs/<slug>/. Removes the runtime dependency on docs.oracle.com when authoring specs and feeds the pattern-lookup scorer. - 110+ cached .drawio / .svg / .png references for offline reuse, plus the OCI Toolkit v24.2 import (kb/diagram/assets/oci-toolkit-drawio). Documentation: - docs/skill/output-formats.md — new "Standard diagram-generation procedure (MANDATORY)" + geometry rules + the new validator entry. - SKILL.md option 2 — references the mandatory procedure. - README.md — describes the spec validator, archcenter_pattern_lookup and description fetcher, and updates the KB-health table. Tooling that backs the procedure (cumulative across recent sessions): tools/archcenter_case_runner.py, archcenter_batch_driver.py, archcenter_zip_downloader.py, drawio_visual_validator.py, drawio_fidelity_eval.py, harvest_drawio_icon.py, import_oci_library.py, oci_pptx_diagram_gen.py, oci_pptx_render.py, refresh_pptx_icon_index.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
@@ -0,0 +1,237 @@
|
||||
# Multicloud data lake integration
|
||||
|
||||
- Source: https://docs.oracle.com/en/solutions/oci-multicloud-datalake/index.html
|
||||
- Date: 2024-03
|
||||
- Type: reference-architecture
|
||||
- Services: data-integration, oic, object-storage, adw, streaming
|
||||
- Tags: data-platform, multicloud, integration
|
||||
|
||||
## Summary (catalog)
|
||||
|
||||
Bring data from AWS/Azure/on-prem into OCI data lake. OCI Data Integration for batch ETL, OIC for app integration with pre-built adapters. Read-only credentials for source systems recommended.
|
||||
|
||||
## Architecture (fetched from source)
|
||||
|
||||
Architecture
|
||||
|
||||
|
||||
|
||||
|
||||
This reference architecture describes how you can bring the data from
|
||||
different cloud providers and on-premises data sources to a data lake hosted in OCI.
|
||||
This architecture covers batch integration, data integration, real time integration
|
||||
and event based integration scenarios.
|
||||
|
||||
|
||||
The following diagram illustrates the data flow for this reference
|
||||
architecture.
|
||||
|
||||
Description of the illustration oci_multicloud_datalake_flow.png
|
||||
|
||||
|
||||
oci-multicloud-datalake-flow-oracle.zip
|
||||
|
||||
|
||||
OCI Data Integration:
|
||||
|
||||
|
||||
- Connects and extracts data from:
|
||||
|
||||
|
||||
- AWS services and Azure services through native
|
||||
adapters.
|
||||
|
||||
- On-premises data sources through private
|
||||
connectivity (FastConnect/VPN).
|
||||
|
||||
- Oracle SaaS applications through BICC
|
||||
connector.
|
||||
|
||||
|
||||
|
||||
- Performs transformation on the extracted data.
|
||||
|
||||
- Loads data into OCI data lake through adapters (ADB/Object
|
||||
Storage).
|
||||
|
||||
|
||||
|
||||
|
||||
Oracle Integration Cloud:
|
||||
|
||||
|
||||
- Receives real time data from various source systems
|
||||
like Oracle SaaS applications/IOT/Streaming services/social
|
||||
media/on-premises systems/other Cloud providers through
|
||||
native adapters.
|
||||
|
||||
- Performs transformation/orchestration logic.
|
||||
|
||||
- Loads data into OCI data lake through adapters (ADB/Object
|
||||
Storage).
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
The following diagram illustrates this reference architecture.
|
||||
|
||||
|
||||
Description of the illustration oci_multicloud_datalake.png
|
||||
|
||||
oci-multicloud-datalake-oracle.zip
|
||||
|
||||
|
||||
Oracle Data Integration Service is used for the following
|
||||
scenarios:
|
||||
|
||||
|
||||
- Consolidating data by capturing data from multiple,
|
||||
heterogeneous source systems and integrating into a single
|
||||
persistent store. This is typically accomplished using
|
||||
extract, transform and load (ETL) routines.
|
||||
|
||||
- Extracting high volume data from the source systems
|
||||
(HDFS, Oracle Autonomous database, MySQL, Oracle Database,
|
||||
Azure Synapse, AWS Redshift, Object Storage, S3, Microsoft
|
||||
SQL, PostgreSQL, and so on) which are hosted in the
|
||||
private/public network (customer on-premises, 3rd party
|
||||
cloud network (Azure VNet, AWS VPC)) and then loaded into
|
||||
the OCI data lake.
|
||||
|
||||
- Extracting the data from Oracle
|
||||
Fusion Cloud Applications through BICC/BI Publisher connector and then loading into
|
||||
the OCI data lake.
|
||||
|
||||
|
||||
- Extracting high volume data from multiple sources
|
||||
with an orchestration pattern.
|
||||
|
||||
- Implementing scheduled (daily, monthly, weekly,
|
||||
monthly, cron expression, and so on) ETL jobs.
|
||||
|
||||
|
||||
|
||||
|
||||
Oracle Integration Cloud (OIC) is used for the following
|
||||
scenarios:
|
||||
|
||||
|
||||
|
||||
|
||||
- Receiving data from Oracle Cloud applications, CRM,
|
||||
E-commerce and on-premises/3rd party cloud applications in real-time
|
||||
and then loading into data lake.
|
||||
|
||||
- Loading the data into data lake from a file (less volume)
|
||||
generated by a data-source.
|
||||
|
||||
- Exposing Oracle Integration Cloud REST APIs to webhook platforms, receiving the data in
|
||||
real-time and loading into the data lake.
|
||||
|
||||
|
||||
- Some IOT platforms (Geotab, CheckSafe, and so on) have webhook
|
||||
fuctionality and send data to any https api for new events so they
|
||||
can connect directly to the API Gateway.
|
||||
|
||||
- Receiving data from social media platforms (Facebook,
|
||||
LinkedIn, Twitter, Slack, and so on) and loading into the OCI data
|
||||
lake.
|
||||
|
||||
|
||||
Oracle API Gateway is used for the following scenarios:
|
||||
|
||||
|
||||
- Publishing OIC APIs and Application APIs with private endpoints
|
||||
that are accessible from within your network or you can
|
||||
expose to the public internet if required. The endpoints
|
||||
support API validation, request and response transformation,
|
||||
CORS, authentication and authorization, and request
|
||||
limiting.
|
||||
|
||||
- Decoupling the security and business logic in API
|
||||
development.
|
||||
|
||||
- Exposing APIs to the restricted sources with security controls
|
||||
which may feed the data to downstream data lake.
|
||||
|
||||
|
||||
|
||||
|
||||
The architecture has the following components:
|
||||
|
||||
|
||||
|
||||
- Region
|
||||
An Oracle Cloud
|
||||
Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).
|
||||
|
||||
|
||||
|
||||
|
||||
- Availability domains
|
||||
Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain is unlikely to affect the other availability domains in the region.
|
||||
|
||||
|
||||
|
||||
- Virtual cloud network (VCN) and subnets
|
||||
A VCN is a customizable, software-defined network that you set up in an Oracle Cloud
|
||||
Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.
|
||||
|
||||
|
||||
|
||||
|
||||
- Integration
|
||||
Oracle Integration is a fully managed service that allows you to integrate your applications, automate processes, gain insight into your business processes, and create visual applications.
|
||||
|
||||
|
||||
|
||||
|
||||
- Oracle Data Integration
|
||||
Oracle Cloud Infrastructure Data
|
||||
Integration is a fully managed, serverless, cloud-native service that extracts, loads, transforms, cleanses, and reshapes data from a variety of data sources into target Oracle Cloud
|
||||
Infrastructure services, such as Autonomous Data
|
||||
Warehouse and Oracle Cloud
|
||||
Infrastructure Object Storage . ETL (extract transform load) leverages fully-managed scale-out processing on Spark, and ELT (extract load transform) leverages full SQL push-down capabilities of the Autonomous Data
|
||||
Warehouse in order to minimize data movement and to improve the time to value for newly ingested data. Users design data integration processes using an intuitive, codeless user interface that optimizes integration flows to generate the most efficient engine and orchestration, automatically allocating and scaling the execution environment. Oracle Cloud Infrastructure Data
|
||||
Integration provides interactive exploration and data preparation and helps data engineers protect against schema drift by defining rules to handle schema changes.
|
||||
|
||||
|
||||
|
||||
|
||||
- Oracle Business
|
||||
Intelligence
|
||||
Cloud Connector
|
||||
The Oracle BI Cloud Connector
|
||||
(BICC) is a useful tool for extracting data from Fusion and
|
||||
for storing it in shared resources like Oracle Universal
|
||||
Content Management (UCM) Server or cloud storage in CSV
|
||||
format.
|
||||
|
||||
|
||||
|
||||
- OIC Connectivity Agent
|
||||
With the OIC connectivity agent, you can
|
||||
create hybrid integrations and exchange messages between
|
||||
applications in private or on-premises networks and Oracle
|
||||
Integration Cloud.
|
||||
|
||||
|
||||
|
||||
- Data Lake
|
||||
A data lake is a scalable,
|
||||
centralized repository that can store raw data and enables
|
||||
an enterprise to store all its data in a cost effective,
|
||||
elastic environment. A data lake provides a flexible storage
|
||||
mechanism for storing raw data.
|
||||
|
||||
|
||||
|
||||
- Object storage
|
||||
Object storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. You can safely and securely store and then retrieve data directly from the internet or from within the cloud platform. You can seamlessly scale storage without experiencing any degradation in performance or service reliability. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access.
|
||||
|
||||
|
||||
|
||||
- Autonomous Database
|
||||
Oracle Cloud
|
||||
Infrastructure Autonomous Database i
|
||||
Reference in New Issue
Block a user