# Multicloud data lake integration

- Source: https://docs.oracle.com/en/solutions/oci-multicloud-datalake/index.html
- Date: 2024-03
- Type: reference-architecture
- Services: data-integration, oic, object-storage, adw, streaming
- Tags: data-platform, multicloud, integration

## Summary (catalog)

Bring data from AWS/Azure/on-prem into OCI data lake. OCI Data Integration for batch ETL, OIC for app integration with pre-built adapters. Read-only credentials for source systems recommended.

## Architecture (fetched from source)

Architecture

 

 
This reference architecture describes how you can bring the data from
 different cloud providers and on-premises data sources to a data lake hosted in OCI.
 This architecture covers batch integration, data integration, real time integration
 and event based integration scenarios.

 
The following diagram illustrates the data flow for this reference
 architecture.
 
 Description of the illustration oci_multicloud_datalake_flow.png 

 
 oci-multicloud-datalake-flow-oracle.zip 

 
OCI Data Integration:
 
 
- Connects and extracts data from:
 
 
- AWS services and Azure services through native
 adapters. 
 
- On-premises data sources through private
 connectivity (FastConnect/VPN). 
 
- Oracle SaaS applications through BICC
 connector. 
 
 
 
- Performs transformation on the extracted data. 
 
- Loads data into OCI data lake through adapters (ADB/Object
 Storage). 
 
 

 
Oracle Integration Cloud:
 
 
- Receives real time data from various source systems
 like Oracle SaaS applications/IOT/Streaming services/social
 media/on-premises systems/other Cloud providers through
 native adapters. 
 
- Performs transformation/orchestration logic. 
 
- Loads data into OCI data lake through adapters (ADB/Object
 Storage). 
 
 
 

 
The following diagram illustrates this reference architecture.

 
 Description of the illustration oci_multicloud_datalake.png 

 oci-multicloud-datalake-oracle.zip 

 
 Oracle Data Integration Service is used for the following
 scenarios:
 
 
- Consolidating data by capturing data from multiple,
 heterogeneous source systems and integrating into a single
 persistent store. This is typically accomplished using
 extract, transform and load (ETL) routines. 
 
- Extracting high volume data from the source systems
 (HDFS, Oracle Autonomous database, MySQL, Oracle Database,
 Azure Synapse, AWS Redshift, Object Storage, S3, Microsoft
 SQL, PostgreSQL, and so on) which are hosted in the
 private/public network (customer on-premises, 3rd party
 cloud network (Azure VNet, AWS VPC)) and then loaded into
 the OCI data lake. 
 
- Extracting the data from Oracle
 Fusion Cloud Applications through BICC/BI Publisher connector and then loading into
 the OCI data lake.
 
 
- Extracting high volume data from multiple sources
 with an orchestration pattern. 
 
- Implementing scheduled (daily, monthly, weekly,
 monthly, cron expression, and so on) ETL jobs. 
 
 

 
 Oracle Integration Cloud (OIC) is used for the following
 scenarios:
 

 
 
- Receiving data from Oracle Cloud applications, CRM,
 E-commerce and on-premises/3rd party cloud applications in real-time
 and then loading into data lake. 
 
- Loading the data into data lake from a file (less volume)
 generated by a data-source. 
 
- Exposing Oracle Integration Cloud REST APIs to webhook platforms, receiving the data in
 real-time and loading into the data lake.
 
 
- Some IOT platforms (Geotab, CheckSafe, and so on) have webhook
 fuctionality and send data to any https api for new events so they
 can connect directly to the API Gateway. 
 
- Receiving data from social media platforms (Facebook,
 LinkedIn, Twitter, Slack, and so on) and loading into the OCI data
 lake. 
 
 
 Oracle API Gateway is used for the following scenarios:
 
 
- Publishing OIC APIs and Application APIs with private endpoints
 that are accessible from within your network or you can
 expose to the public internet if required. The endpoints
 support API validation, request and response transformation,
 CORS, authentication and authorization, and request
 limiting. 
 
- Decoupling the security and business logic in API
 development. 
 
- Exposing APIs to the restricted sources with security controls
 which may feed the data to downstream data lake. 
 
 

 
The architecture has the following components:

 
 
- Region 
An Oracle Cloud
 Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).
 

 
 
- Availability domains 
Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain is unlikely to affect the other availability domains in the region.

 
 
- Virtual cloud network (VCN) and subnets 
A VCN is a customizable, software-defined network that you set up in an Oracle Cloud
 Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private. 
 

 
 
- Integration 
 Oracle Integration is a fully managed service that allows you to integrate your applications, automate processes, gain insight into your business processes, and create visual applications.
 

 
 
- Oracle Data Integration 
 Oracle Cloud Infrastructure Data
 Integration is a fully managed, serverless, cloud-native service that extracts, loads, transforms, cleanses, and reshapes data from a variety of data sources into target Oracle Cloud
 Infrastructure services, such as Autonomous Data
 Warehouse and Oracle Cloud
 Infrastructure Object Storage . ETL (extract transform load) leverages fully-managed scale-out processing on Spark, and ELT (extract load transform) leverages full SQL push-down capabilities of the Autonomous Data
 Warehouse in order to minimize data movement and to improve the time to value for newly ingested data. Users design data integration processes using an intuitive, codeless user interface that optimizes integration flows to generate the most efficient engine and orchestration, automatically allocating and scaling the execution environment. Oracle Cloud Infrastructure Data
 Integration provides interactive exploration and data preparation and helps data engineers protect against schema drift by defining rules to handle schema changes. 
 

 
 
- Oracle Business
 Intelligence 
 Cloud Connector 
The Oracle BI Cloud Connector
 (BICC) is a useful tool for extracting data from Fusion and
 for storing it in shared resources like Oracle Universal
 Content Management (UCM) Server or cloud storage in CSV
 format.

 
 
- OIC Connectivity Agent 
With the OIC connectivity agent, you can
 create hybrid integrations and exchange messages between
 applications in private or on-premises networks and Oracle
 Integration Cloud.

 
 
- Data Lake 
A data lake is a scalable,
 centralized repository that can store raw data and enables
 an enterprise to store all its data in a cost effective,
 elastic environment. A data lake provides a flexible storage
 mechanism for storing raw data.

 
 
- Object storage 
 Object storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. You can safely and securely store and then retrieve data directly from the internet or from within the cloud platform. You can seamlessly scale storage without experiencing any degradation in performance or service reliability. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access.

 
 
- Autonomous Database 
 Oracle Cloud
 Infrastructure Autonomous Database i
