# Field Knowledge: Real-World Gotchas and Lessons Learned # From actual OCI deployments — things the documentation doesn't emphasize gotchas: - id: FK-001 area: "ADB-S Auto-Scaling" severity: HIGH description: | ADB-S auto-scaling takes 2-3 minutes to activate. If you size base OCPUs for average load (P50), short spikes will hit before scale-out completes. Size base OCPUs for P75 sustained load. recommendation: "Base OCPU = P75 CPU usage, not P50" - id: FK-002 area: "ADB-S Private Endpoint" severity: HIGH description: | ADB-S with public endpoint allows connections from the internet to the DB listener. Always use private endpoints for production. This requires a VCN subnet and Service Gateway. recommendation: "Always use private endpoints. Never public for production." - id: FK-003 area: "FastConnect Redundancy" severity: HIGH description: | A single FastConnect virtual circuit is a single point of failure. For production, deploy 2 virtual circuits on diverse physical paths. Also configure IPSec VPN as a backup path. recommendation: "2x FastConnect (diverse paths) + VPN backup" - id: FK-004 area: "Service Limits" severity: MEDIUM description: | OCI has default service limits per tenancy that are often too low for production deployments. Request limit increases BEFORE go-live, as approval can take 1-3 business days. recommendation: "Submit service limit increase requests during planning phase" - id: FK-005 area: "Block Volume Performance" severity: MEDIUM description: | Block Volume IOPS scale with volume size. A 50 GB Balanced volume only provides 3,000 IOPS. If you need 25,000 IOPS, you need at least 417 GB (even if your data is only 50 GB). recommendation: "Size volumes for IOPS, not just capacity. Over-provision if needed." - id: FK-006 area: "OKE LoadBalancer Annotations" severity: LOW description: | OKE creates a 10 Mbps Flexible Load Balancer by default for LoadBalancer-type Kubernetes services. For production, set the bandwidth annotation explicitly. recommendation: | Set annotation: service.beta.kubernetes.io/oci-load-balancer-shape: "flexible" And: service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "100" - id: FK-007 area: "Terraform State" severity: MEDIUM description: | Don't store Terraform state locally or in git. Use OCI Object Storage with versioning as the remote backend. Enable state locking. recommendation: "Use OCI Object Storage as Terraform remote backend with state locking." - id: FK-008 area: "Cross-Region Data Transfer" severity: MEDIUM description: | Cross-region data transfer is not free. Data Guard redo transport, Object Storage replication, and cross-region backup copy all incur egress charges. Factor this into DR cost estimates. recommendation: "Include cross-region egress in DR cost estimates" - id: FK-009 area: "Identity Domain Limits" severity: LOW description: | Free tier Identity Domain supports up to 2,000 users. For larger deployments, upgrade to Premium Identity Domain. recommendation: "Check user count against Identity Domain tier limits" - id: FK-010 area: "Compartment Depth" severity: LOW description: | Compartments can be nested up to 6 levels deep. In practice, 3-4 levels is the sweet spot. Deeper nesting makes IAM policies harder to manage. recommendation: "Limit compartment depth to 3-4 levels" - id: FK-011 area: "VCN CIDR Planning" severity: HIGH description: | VCN CIDRs cannot be changed after creation (you can add secondary CIDRs but not modify the primary). Plan CIDR ranges carefully, especially when on-premises integration is required. recommendation: "Use /16 per VCN, document all CIDRs, avoid overlap with on-prem" - id: FK-012 area: "Data Guard Switchover vs Failover" severity: HIGH description: | Switchover is planned (zero data loss, both DBs running). Failover is unplanned (potential data loss with async). Ensure runbooks distinguish between the two and teams practice both. recommendation: "Document and practice both switchover (planned) and failover (unplanned)" - id: FK-013 area: "TAC Replay Limitations" severity: MEDIUM description: | Transparent Application Continuity (TAC) cannot replay operations that have external side effects: UTL_HTTP calls, DBMS_PIPE, NOCACHE sequences, and certain PL/SQL constructs. These cause replay to abort and the application sees the error. Audit application code before relying on TAC. recommendation: "Audit application for non-replayable operations before enabling TAC" - id: FK-014 area: "ADB-S Auto Indexing" severity: LOW description: | ADB-S Auto Indexing creates indexes with VISIBILITY=INVISIBLE (HIDDEN). They exist and are used by the optimizer but don't appear in standard DBA_INDEXES views unless you filter for invisible. This confuses DBAs who think no indexes were created. recommendation: "Query DBA_INDEXES with VISIBILITY='INVISIBLE' to see auto-created indexes" - id: FK-015 area: "Database Enterprise Patching (DEP)" severity: MEDIUM description: | DEP provisioning can take 30-60 minutes for DBCS/ExaCS. Plan maintenance windows accordingly. DEP also requires Fleet Management setup which has its own prerequisites (OS Management Hub agent, etc.). recommendation: "Set up DEP and Fleet Management early; plan for 60-minute patching windows" - id: FK-016 area: "ExaCS Minimum Cost" severity: HIGH description: | ExaCS requires dedicated Exadata infrastructure. Minimum configuration (quarter rack) starts at ~$6,000/month before database licensing. Don't recommend for workloads under 16 OCPUs sustained. recommendation: "Use ADB-S or DBCS for workloads < 16 OCPUs; ExaCS only for isolation/RAC/high IOPS" - id: FK-017 area: "Cross-Region ADG is Async Only" severity: HIGH description: | Cross-region Autonomous Data Guard uses asynchronous replication only. RPO is always > 0 (typically seconds). For zero RPO you need same-region standby or ADB-D with local synchronous Data Guard. recommendation: "Set customer expectations: cross-region DR = RPO > 0 (seconds, not zero)" - id: FK-018 area: "OKE VCN-Native Pod Networking" severity: MEDIUM description: | VCN-Native Pod Networking (OCI CNI) assigns VCN IPs directly to pods. This enables NSG attachment to pods but consumes subnet IP addresses rapidly. A /24 subnet supports ~250 pods. Size subnets for pod count, not node count. recommendation: "Use /21 or larger subnets for pod networking; calculate IPs from max pod count" - id: FK-019 area: "Object Storage Eventual Consistency" severity: MEDIUM description: | OCI Object Storage provides strong read-after-write consistency for PUTs of new objects, but LIST operations may show eventual consistency. If your application lists then reads immediately after upload, the new object may not appear in the list yet. recommendation: "Don't rely on LIST immediately after PUT; use specific GET for confirmation" - id: FK-020 area: "Database@AWS — ADB-S NOT available" severity: HIGH description: | Oracle Database@AWS only supports ADB on Dedicated Exadata Infrastructure (ADB-D), NOT ADB-S Serverless. Customers expecting serverless auto-scaling and pay-per-query pricing will be disappointed. If ADB-S is required, the architecture must use native OCI with FastConnect from AWS — which adds ~1-2ms RTT vs the ~0.2-0.4ms of Database@AWS co-located infra. recommendation: | Clarify ADB-S vs ADB-D early in discovery. If P95 <1ms AND serverless are both hard requirements, they are mutually exclusive with current Database@AWS offering. Trade-off: co-located ADB-D (low latency) vs native OCI ADB-S (serverless) via FastConnect (higher latency). - id: FK-021 area: "Database@AWS — Same-AZ placement is critical" severity: HIGH description: | Database@AWS latency drops from ~0.2-0.4ms (same AZ) to ~1ms+ (cross-AZ). If app EC2 and ODB network are in different AZs, you lose the latency benefit that justifies co-location. Transit Gateway attachment must also be in the same AZ as the ODB network. recommendation: | Always deploy app and ODB network in the same AZ. Document AZ choice early — VM clusters cannot be moved between ODB networks after creation. - id: FK-022 area: "Flex Shape Memory Ratio" severity: LOW description: | VM.Standard.E4.Flex allows 1-64 GB per OCPU (default 16 GB). For memory-intensive workloads, you can increase memory without adding OCPUs. But memory is billed separately — verify it's cheaper than adding OCPUs with default memory. recommendation: "Compare cost of extra memory vs extra OCPUs for memory-intensive workloads" - id: FK-024 area: "Exadata X11M — Minimum Configuration (ADB-D and ExaCS)" severity: HIGH description: | Exadata X11M applies the same minimum configuration for both ADB-D and ExaCS: 2 DB servers + 3 storage servers. It is not possible to provision a single DB server. This applies to X8M, X9M, and X11M generations. Elastic range: 2–32 DB servers, 3–64 storage servers. The tenancy service limits for X11M must be set to at least DB=2, Storage=3 before provisioning. recommendation: "Always start X11M sizing at 2 DB servers + 3 storage servers minimum. Applies to both ADB-D and ExaCS." source: "Verified — Oracle docs (ExaDB-D X11M datasheet, ADB-D X11M datasheet, OCI docs)" references: - "https://www.oracle.com/a/ocom/docs/engineered-systems/exadata/exadb-d-x11m-ds.pdf" - "https://docs.oracle.com/en/cloud/paas/autonomous-database/dedicated/adbde/" - id: FK-023 area: "GPU Bare Metal — Preemptible Availability" severity: HIGH description: | OCI Preemptible instances (~50% off) are technically supported for BM shapes, but for high-demand GPU bare metal shapes (BM.GPU.H100, BM.GPU.H200, BM.GPU.B200, BM.GPU.B300) preemptible capacity is practically unavailable. These shapes are sold almost entirely via reservations or Universal Credits commitments. Additionally, preemptible instances can be reclaimed with only 30 seconds notice, making them operationally unviable for multi-day or multi-week training runs. recommendation: | Do NOT model GPU BM cost savings based on preemptible pricing. Use Universal Credits 1yr (~35% off) or 3yr (~55% off) for realistic estimates. If the workload is short batch jobs tolerant to interruptions, ask your PDM about spot/preemptible availability before committing to that model. source: "Field experience — Diego, OCI SA"