# Multicloud data lake integration - Source: https://docs.oracle.com/en/solutions/oci-multicloud-datalake/index.html - Date: 2024-03 - Type: reference-architecture - Services: data-integration, oic, object-storage, adw, streaming - Tags: data-platform, multicloud, integration ## Summary (catalog) Bring data from AWS/Azure/on-prem into OCI data lake. OCI Data Integration for batch ETL, OIC for app integration with pre-built adapters. Read-only credentials for source systems recommended. ## Architecture (fetched from source) Architecture This reference architecture describes how you can bring the data from different cloud providers and on-premises data sources to a data lake hosted in OCI. This architecture covers batch integration, data integration, real time integration and event based integration scenarios. The following diagram illustrates the data flow for this reference architecture. Description of the illustration oci_multicloud_datalake_flow.png oci-multicloud-datalake-flow-oracle.zip OCI Data Integration: - Connects and extracts data from: - AWS services and Azure services through native adapters. - On-premises data sources through private connectivity (FastConnect/VPN). - Oracle SaaS applications through BICC connector. - Performs transformation on the extracted data. - Loads data into OCI data lake through adapters (ADB/Object Storage). Oracle Integration Cloud: - Receives real time data from various source systems like Oracle SaaS applications/IOT/Streaming services/social media/on-premises systems/other Cloud providers through native adapters. - Performs transformation/orchestration logic. - Loads data into OCI data lake through adapters (ADB/Object Storage). The following diagram illustrates this reference architecture. Description of the illustration oci_multicloud_datalake.png oci-multicloud-datalake-oracle.zip Oracle Data Integration Service is used for the following scenarios: - Consolidating data by capturing data from multiple, heterogeneous source systems and integrating into a single persistent store. This is typically accomplished using extract, transform and load (ETL) routines. - Extracting high volume data from the source systems (HDFS, Oracle Autonomous database, MySQL, Oracle Database, Azure Synapse, AWS Redshift, Object Storage, S3, Microsoft SQL, PostgreSQL, and so on) which are hosted in the private/public network (customer on-premises, 3rd party cloud network (Azure VNet, AWS VPC)) and then loaded into the OCI data lake. - Extracting the data from Oracle Fusion Cloud Applications through BICC/BI Publisher connector and then loading into the OCI data lake. - Extracting high volume data from multiple sources with an orchestration pattern. - Implementing scheduled (daily, monthly, weekly, monthly, cron expression, and so on) ETL jobs. Oracle Integration Cloud (OIC) is used for the following scenarios: - Receiving data from Oracle Cloud applications, CRM, E-commerce and on-premises/3rd party cloud applications in real-time and then loading into data lake. - Loading the data into data lake from a file (less volume) generated by a data-source. - Exposing Oracle Integration Cloud REST APIs to webhook platforms, receiving the data in real-time and loading into the data lake. - Some IOT platforms (Geotab, CheckSafe, and so on) have webhook fuctionality and send data to any https api for new events so they can connect directly to the API Gateway. - Receiving data from social media platforms (Facebook, LinkedIn, Twitter, Slack, and so on) and loading into the OCI data lake. Oracle API Gateway is used for the following scenarios: - Publishing OIC APIs and Application APIs with private endpoints that are accessible from within your network or you can expose to the public internet if required. The endpoints support API validation, request and response transformation, CORS, authentication and authorization, and request limiting. - Decoupling the security and business logic in API development. - Exposing APIs to the restricted sources with security controls which may feed the data to downstream data lake. The architecture has the following components: - Region An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents). - Availability domains Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain is unlikely to affect the other availability domains in the region. - Virtual cloud network (VCN) and subnets A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private. - Integration Oracle Integration is a fully managed service that allows you to integrate your applications, automate processes, gain insight into your business processes, and create visual applications. - Oracle Data Integration Oracle Cloud Infrastructure Data Integration is a fully managed, serverless, cloud-native service that extracts, loads, transforms, cleanses, and reshapes data from a variety of data sources into target Oracle Cloud Infrastructure services, such as Autonomous Data Warehouse and Oracle Cloud Infrastructure Object Storage . ETL (extract transform load) leverages fully-managed scale-out processing on Spark, and ELT (extract load transform) leverages full SQL push-down capabilities of the Autonomous Data Warehouse in order to minimize data movement and to improve the time to value for newly ingested data. Users design data integration processes using an intuitive, codeless user interface that optimizes integration flows to generate the most efficient engine and orchestration, automatically allocating and scaling the execution environment. Oracle Cloud Infrastructure Data Integration provides interactive exploration and data preparation and helps data engineers protect against schema drift by defining rules to handle schema changes. - Oracle Business Intelligence Cloud Connector The Oracle BI Cloud Connector (BICC) is a useful tool for extracting data from Fusion and for storing it in shared resources like Oracle Universal Content Management (UCM) Server or cloud storage in CSV format. - OIC Connectivity Agent With the OIC connectivity agent, you can create hybrid integrations and exchange messages between applications in private or on-premises networks and Oracle Integration Cloud. - Data Lake A data lake is a scalable, centralized repository that can store raw data and enables an enterprise to store all its data in a cost effective, elastic environment. A data lake provides a flexible storage mechanism for storing raw data. - Object storage Object storage provides quick access to large amounts of structured and unstructured data of any content type, including database backups, analytic data, and rich content such as images and videos. You can safely and securely store and then retrieve data directly from the internet or from within the cloud platform. You can seamlessly scale storage without experiencing any degradation in performance or service reliability. Use standard storage for "hot" storage that you need to access quickly, immediately, and frequently. Use archive storage for "cold" storage that you retain for long periods of time and seldom or rarely access. - Autonomous Database Oracle Cloud Infrastructure Autonomous Database i