diegoecab/oci-deal-accelerator
1
1
Fork 2
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e3ad1e3884740c2f12914c5ad1d43548ec33a932
oci-deal-accelerator/kb/diagram/assets/archcenter-refs/analytics-pipeline-db-at-azure/_description.md
root b30a4f0d32 Diagram generation: ref-arch-driven procedure + spec validator + KB enrichment 
The diagram path now follows a documented standard procedure (lookup
the closest Oracle Architecture Center reference → confirm components
→ author absolute_layout → spec validator → render → visually verify)
and ships persistent guardrails so layout regressions can't recur.

Persistent procedure changes (apply to all users, all sessions):
- tools/diagram_spec_validator.py — geometry checks (CONTAINER_TOO_THIN,
  CONTAINER_PADDING_VIOLATION, LABEL_OVERFLOW_PARENT) run BEFORE either
  renderer (drawio + PPTX). Catches the subnet-collapse / label-overflow
  bugs that the post-render drawio validator missed.
- tools/oci_diagram_gen.py + tools/oci_pptx_diagram_gen.py — call the
  spec validator before emitting any output. Adds mysql / mysql_heatwave
  type aliases.
- tools/archcenter_pattern_lookup.py — scores against cached page
  descriptions (not just the 1-line summary), supports --queries for
  multi-fragment composition, and applies synonym expansion via
  kb/architecture-center/synonyms.yaml so "LB HA cross AD" matches
  "load balancer high availability availability domain".
- kb/architecture-center/synonyms.yaml — canonical synonym table
  (load balancer, autonomous database, data guard, …) used by the
  lookup scorer.

KB enrichment:
- tools/archcenter_description_fetcher.py + 121 cached _description.md
  under kb/diagram/assets/archcenter-refs/<slug>/. Removes the runtime
  dependency on docs.oracle.com when authoring specs and feeds the
  pattern-lookup scorer.
- 110+ cached .drawio / .svg / .png references for offline reuse,
  plus the OCI Toolkit v24.2 import (kb/diagram/assets/oci-toolkit-drawio).

Documentation:
- docs/skill/output-formats.md — new "Standard diagram-generation
  procedure (MANDATORY)" + geometry rules + the new validator entry.
- SKILL.md option 2 — references the mandatory procedure.
- README.md — describes the spec validator, archcenter_pattern_lookup
  and description fetcher, and updates the KB-health table.

Tooling that backs the procedure (cumulative across recent sessions):
tools/archcenter_case_runner.py, archcenter_batch_driver.py,
archcenter_zip_downloader.py, drawio_visual_validator.py,
drawio_fidelity_eval.py, harvest_drawio_icon.py, import_oci_library.py,
oci_pptx_diagram_gen.py, oci_pptx_render.py, refresh_pptx_icon_index.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 21:15:21 -03:00

8.3 KiB
Raw Blame History

Deploy a Data Lake leveraging Power BI on Oracle Database@Azure

Summary (catalog)

Analytics pipeline combining ADB-S on Database@Azure with Power BI. Data ingestion to Object Storage, transformation in ADB-S, visualization via Power BI DirectQuery or Import mode.

Architecture (fetched from source)

Deploy a Data Lake Leveraging Power BI on Oracle Database@Azure

Previous Next JavaScript must be enabled to correctly display this content

  • Deploy a Data Lake leveraging Power BI on Oracle Database@Azure

  • Deploy a Data Lake Leveraging Power BI on Oracle Database@Azure

Deploy a Data Lake Leveraging Power BI on Oracle Database@Azure

Many businesses leverage Microsoft Power BI with data lakes on Microsoft Azure to derive actionable business insights.

You can expand these capabilities by using a medallion architecture that includes Azure Data Factory, Azure Data Lake Storage, Azure Compute, Oracle Database@Azure (either a fully managed Oracle Autonomous Database or a co-managed Oracle Exadata Database Service instance), and Power BI to address several key data challenges faced by customers:

  • Data Silos and Integration : Azure Data Factory ingests data from diverse sources into a unified data lake, breaking down silos and providing a single source of truth.

  • Data Quality and Consistency : Autonomous Data Warehouse in the Curation Layer ensures clean, consistent, and high-quality data through deduplication and quality rules, reducing errors and enhancing decision-making.

  • Scalability and Performance : Azure's scalable compute resources and Autonomous Data Warehouse 's serverless architecture or Oracle Exadata Database Service handle large-scale data processing efficiently, while maintaining optimal performance as data volumes and user adoption (concurrency) grow.

  • Complex Transformations : Azure Compute and Autonomous Data Warehouse or Oracle Exadata Database Service perform complex transformations and analytics efficiently, reducing processing time and focusing on insights.

  • Cost Management : The serverless and pay-as-you-go models for Azure services and Autonomous Data Warehouse or Oracle Exadata Database Service optimize costs, ensuring that you only pay for what you use.

  • Data Governance and Compliance : Structured data management layers facilitate better governance, traceability, and regulatory compliance.

  • Built-in Analytics : Users are able to apply analytics directly to their data by using built-in features such as artificial intelligence (AI), machine learning (ML), graph, spatial, and text analytics.

Typical use cases include:

  • Retail Analytics : Integrates data from online sales, in-store transactions, and customer feedback, optimizing inventory and marketing strategies.

  • Financial Services : Analyzes transaction data for fraud detection and regulatory compliance, mitigating risks.

  • Healthcare Analytics : Integrates patient data from EHRs, lab results, and wearable devices, improving patient care and health management.

This architecture enables enterprise customers across industries to leverage data effectively to empower their business users to make informed decisions to drive better business outcomes.

Logical Architecture

The analytical data lake can ingest data from multiple sources and can provide business insights by using Power BI running on Microsoft Azure.

  • Data Sources: The analytical data lake can ingest data from multiple sources. Azure Data Factory can ingest data from Microsoft SQL Server and Azure Blob Storage. Oracle Database@Azure can ingest data from Oracle Cloud ERP , Oracle Cloud Infrastructure Object Storage , Azure Cosmos Database, Azure SQL Database, various types of table storage data (Azure, PostgresSQL, Azure MariaDB), and other types of on-premises relational databases.

  • Data Tier: Oracle Database@Azure ingests source data from Azure Data Lake Storage in conjunction with Azure Data Factory.

  • Consumption Tier: Oracle Database@Azure provides insights to Microsoft Power BI running on Microsoft Azure.

The following diagram illustrates the functional architecture:

Description of the illustration data-lake-db-azure-process.png

data-lake-db-azure-process-oracle.zip

Medallion Architecture

This section demonstrates how you can deploy Oracle Database@Azure as the data warehouse within the Azure medallion architecture.

The medallion architecture is a data management framework that structures data handling in a data lakehouse into distinct stages (bronze, silver, and gold), representing the different stages of data processing:

  • Bronze stage: Data from various sources is ingested, validated, and curated.

  • Silver stage: The data is stored and processed for analytics and reporting.

  • Gold stage: Refined data is delivered for analysis and reporting.

The following diagram illustrates the architecture:

Description of the illustration data-lake-db-azure-medallion.png

data-lake-db-azure-medallion-oracle.zip

The medallion stages are further divided into the following deployment areas:

  • Ingestion Framework: Ingests data from various data sources using Azure Data Factory. Raw data is stored in Azure Data Lake Storage Gen 2 and Delta Lake. This framework ensures data consistency and accuracy across source and sink systems. This framework constitutes a robust set of scripts to ensure quality by using audit, balance and control mechanisms across platforms.

  • Validation: Raw data is ingested into Oracle Autonomous Data Warehouse Serverless or Oracle Exadata Database Service for deduplication and data quality check. This workflow performs basic cleansing masking of PII and PHI data along with validation of raw files through a rules-driven framework to perform schema checks. The validation framework can be implemented using Azure Data Factory.

  • Rejection Workflow: Any record that is rejected during the ingestion stage due to validation errors or other processing errors is staged on a separate Azure Data Lake Storage path. Automated email notifications using Logic App are sent to the support team based on defined software license agreements (SLAs). Standardized data remains in Oracle Autonomous Data Warehouse Serverless or Oracle Exadata Database Service .

  • Orchestration: A scheduling system manages data processing jobs, scheduling, and job dependencies. Azure Data Factory can be used for the orchestration of ETL jobs. The Orchestration stage includes Oracle Autonomous Data Warehouse Serverless or Oracle Exadata Database Service , Delta Lake, and Azure Data Lake Storage Gen 2.

  • Reporting/Analytics: The reporting stage includes Power BI and data services such as external feeds and data monetization.

The architecture has the following infrastructure components:

  • Region An Azure region is a geographical area in which one or more physical Azure data centers, called availability zones, reside. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).

Azure and OCI regions are localized geographic areas. For Oracle Database@Azure , an Azure region is connected to an OCI region, with availability zones (AZs) in Azure connected to availability domains (ADs) in OCI. Azure and OCI region pairs are selected to minimize distance and latency.

  • Availability zone An availability zone is a physically separate data center within a region designed to be available and fault-tolerant. Availability zones are close enough to have low-latency connections to other availability zones.

  • Virtual network (VNet) and subnet A VNet is a virtual network that you define in Azure. A VNet can have multiple non-overlapping CIDR blocks subnets that you can add after your create the VNet. You can segment a VNet into subnets, which can be scoped to a region or to an availability zones. Each subnet consists of a contiguous range of addresses that

Reference in New Issue View Git Blame Copy Permalink