initial version

kb/field-knowledge/gotchas.yaml

@@ -0,0 +1,247 @@
+# Field Knowledge: Real-World Gotchas and Lessons Learned
+# From actual OCI deployments — things the documentation doesn't emphasize
+
+gotchas:
+  - id: FK-001
+    area: "ADB-S Auto-Scaling"
+    severity: HIGH
+    description: |
+      ADB-S auto-scaling takes 2-3 minutes to activate. If you size base OCPUs
+      for average load (P50), short spikes will hit before scale-out completes.
+      Size base OCPUs for P75 sustained load.
+    recommendation: "Base OCPU = P75 CPU usage, not P50"
+
+  - id: FK-002
+    area: "ADB-S Private Endpoint"
+    severity: HIGH
+    description: |
+      ADB-S with public endpoint allows connections from the internet to the
+      DB listener. Always use private endpoints for production. This requires
+      a VCN subnet and Service Gateway.
+    recommendation: "Always use private endpoints. Never public for production."
+
+  - id: FK-003
+    area: "FastConnect Redundancy"
+    severity: HIGH
+    description: |
+      A single FastConnect virtual circuit is a single point of failure.
+      For production, deploy 2 virtual circuits on diverse physical paths.
+      Also configure IPSec VPN as a backup path.
+    recommendation: "2x FastConnect (diverse paths) + VPN backup"
+
+  - id: FK-004
+    area: "Service Limits"
+    severity: MEDIUM
+    description: |
+      OCI has default service limits per tenancy that are often too low for
+      production deployments. Request limit increases BEFORE go-live, as
+      approval can take 1-3 business days.
+    recommendation: "Submit service limit increase requests during planning phase"
+
+  - id: FK-005
+    area: "Block Volume Performance"
+    severity: MEDIUM
+    description: |
+      Block Volume IOPS scale with volume size. A 50 GB Balanced volume only
+      provides 3,000 IOPS. If you need 25,000 IOPS, you need at least 417 GB
+      (even if your data is only 50 GB).
+    recommendation: "Size volumes for IOPS, not just capacity. Over-provision if needed."
+
+  - id: FK-006
+    area: "OKE LoadBalancer Annotations"
+    severity: LOW
+    description: |
+      OKE creates a 10 Mbps Flexible Load Balancer by default for
+      LoadBalancer-type Kubernetes services. For production, set the
+      bandwidth annotation explicitly.
+    recommendation: |
+      Set annotation: service.beta.kubernetes.io/oci-load-balancer-shape: "flexible"
+      And: service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "100"
+
+  - id: FK-007
+    area: "Terraform State"
+    severity: MEDIUM
+    description: |
+      Don't store Terraform state locally or in git. Use OCI Object Storage
+      with versioning as the remote backend. Enable state locking.
+    recommendation: "Use OCI Object Storage as Terraform remote backend with state locking."
+
+  - id: FK-008
+    area: "Cross-Region Data Transfer"
+    severity: MEDIUM
+    description: |
+      Cross-region data transfer is not free. Data Guard redo transport,
+      Object Storage replication, and cross-region backup copy all incur
+      egress charges. Factor this into DR cost estimates.
+    recommendation: "Include cross-region egress in DR cost estimates"
+
+  - id: FK-009
+    area: "Identity Domain Limits"
+    severity: LOW
+    description: |
+      Free tier Identity Domain supports up to 2,000 users. For larger
+      deployments, upgrade to Premium Identity Domain.
+    recommendation: "Check user count against Identity Domain tier limits"
+
+  - id: FK-010
+    area: "Compartment Depth"
+    severity: LOW
+    description: |
+      Compartments can be nested up to 6 levels deep. In practice, 3-4 levels
+      is the sweet spot. Deeper nesting makes IAM policies harder to manage.
+    recommendation: "Limit compartment depth to 3-4 levels"
+
+  - id: FK-011
+    area: "VCN CIDR Planning"
+    severity: HIGH
+    description: |
+      VCN CIDRs cannot be changed after creation (you can add secondary CIDRs
+      but not modify the primary). Plan CIDR ranges carefully, especially
+      when on-premises integration is required.
+    recommendation: "Use /16 per VCN, document all CIDRs, avoid overlap with on-prem"
+
+  - id: FK-012
+    area: "Data Guard Switchover vs Failover"
+    severity: HIGH
+    description: |
+      Switchover is planned (zero data loss, both DBs running). Failover is
+      unplanned (potential data loss with async). Ensure runbooks distinguish
+      between the two and teams practice both.
+    recommendation: "Document and practice both switchover (planned) and failover (unplanned)"
+
+  - id: FK-013
+    area: "TAC Replay Limitations"
+    severity: MEDIUM
+    description: |
+      Transparent Application Continuity (TAC) cannot replay operations that
+      have external side effects: UTL_HTTP calls, DBMS_PIPE, NOCACHE sequences,
+      and certain PL/SQL constructs. These cause replay to abort and the
+      application sees the error. Audit application code before relying on TAC.
+    recommendation: "Audit application for non-replayable operations before enabling TAC"
+
+  - id: FK-014
+    area: "ADB-S Auto Indexing"
+    severity: LOW
+    description: |
+      ADB-S Auto Indexing creates indexes with VISIBILITY=INVISIBLE (HIDDEN).
+      They exist and are used by the optimizer but don't appear in standard
+      DBA_INDEXES views unless you filter for invisible. This confuses DBAs
+      who think no indexes were created.
+    recommendation: "Query DBA_INDEXES with VISIBILITY='INVISIBLE' to see auto-created indexes"
+
+  - id: FK-015
+    area: "Database Enterprise Patching (DEP)"
+    severity: MEDIUM
+    description: |
+      DEP provisioning can take 30-60 minutes for DBCS/ExaCS. Plan maintenance
+      windows accordingly. DEP also requires Fleet Management setup which has
+      its own prerequisites (OS Management Hub agent, etc.).
+    recommendation: "Set up DEP and Fleet Management early; plan for 60-minute patching windows"
+
+  - id: FK-016
+    area: "ExaCS Minimum Cost"
+    severity: HIGH
+    description: |
+      ExaCS requires dedicated Exadata infrastructure. Minimum configuration
+      (quarter rack) starts at ~$6,000/month before database licensing.
+      Don't recommend for workloads under 16 OCPUs sustained.
+    recommendation: "Use ADB-S or DBCS for workloads < 16 OCPUs; ExaCS only for isolation/RAC/high IOPS"
+
+  - id: FK-017
+    area: "Cross-Region ADG is Async Only"
+    severity: HIGH
+    description: |
+      Cross-region Autonomous Data Guard uses asynchronous replication only.
+      RPO is always > 0 (typically seconds). For zero RPO you need same-region
+      standby or ADB-D with local synchronous Data Guard.
+    recommendation: "Set customer expectations: cross-region DR = RPO > 0 (seconds, not zero)"
+
+  - id: FK-018
+    area: "OKE VCN-Native Pod Networking"
+    severity: MEDIUM
+    description: |
+      VCN-Native Pod Networking (OCI CNI) assigns VCN IPs directly to pods.
+      This enables NSG attachment to pods but consumes subnet IP addresses
+      rapidly. A /24 subnet supports ~250 pods. Size subnets for pod count,
+      not node count.
+    recommendation: "Use /21 or larger subnets for pod networking; calculate IPs from max pod count"
+
+  - id: FK-019
+    area: "Object Storage Eventual Consistency"
+    severity: MEDIUM
+    description: |
+      OCI Object Storage provides strong read-after-write consistency for PUTs
+      of new objects, but LIST operations may show eventual consistency.
+      If your application lists then reads immediately after upload, the new
+      object may not appear in the list yet.
+    recommendation: "Don't rely on LIST immediately after PUT; use specific GET for confirmation"
+
+  - id: FK-020
+    area: "Database@AWS — ADB-S NOT available"
+    severity: HIGH
+    description: |
+      Oracle Database@AWS only supports ADB on Dedicated Exadata Infrastructure
+      (ADB-D), NOT ADB-S Serverless. Customers expecting serverless auto-scaling
+      and pay-per-query pricing will be disappointed. If ADB-S is required,
+      the architecture must use native OCI with FastConnect from AWS — which
+      adds ~1-2ms RTT vs the ~0.2-0.4ms of Database@AWS co-located infra.
+    recommendation: |
+      Clarify ADB-S vs ADB-D early in discovery. If P95 <1ms AND serverless
+      are both hard requirements, they are mutually exclusive with current
+      Database@AWS offering. Trade-off: co-located ADB-D (low latency) vs
+      native OCI ADB-S (serverless) via FastConnect (higher latency).
+
+  - id: FK-021
+    area: "Database@AWS — Same-AZ placement is critical"
+    severity: HIGH
+    description: |
+      Database@AWS latency drops from ~0.2-0.4ms (same AZ) to ~1ms+ (cross-AZ).
+      If app EC2 and ODB network are in different AZs, you lose the latency
+      benefit that justifies co-location. Transit Gateway attachment must also
+      be in the same AZ as the ODB network.
+    recommendation: |
+      Always deploy app and ODB network in the same AZ. Document AZ choice
+      early — VM clusters cannot be moved between ODB networks after creation.
+
+  - id: FK-022
+    area: "Flex Shape Memory Ratio"
+    severity: LOW
+    description: |
+      VM.Standard.E4.Flex allows 1-64 GB per OCPU (default 16 GB). For
+      memory-intensive workloads, you can increase memory without adding OCPUs.
+      But memory is billed separately — verify it's cheaper than adding OCPUs
+      with default memory.
+    recommendation: "Compare cost of extra memory vs extra OCPUs for memory-intensive workloads"
+
+  - id: FK-024
+    area: "Exadata X11M — Minimum Configuration (ADB-D and ExaCS)"
+    severity: HIGH
+    description: |
+      Exadata X11M applies the same minimum configuration for both ADB-D and ExaCS:
+      2 DB servers + 3 storage servers. It is not possible to provision a single DB
+      server. This applies to X8M, X9M, and X11M generations.
+      Elastic range: 2–32 DB servers, 3–64 storage servers.
+      The tenancy service limits for X11M must be set to at least DB=2, Storage=3
+      before provisioning.
+    recommendation: "Always start X11M sizing at 2 DB servers + 3 storage servers minimum. Applies to both ADB-D and ExaCS."
+    source: "Verified — Oracle docs (ExaDB-D X11M datasheet, ADB-D X11M datasheet, OCI docs)"
+    references:
+      - "https://www.oracle.com/a/ocom/docs/engineered-systems/exadata/exadb-d-x11m-ds.pdf"
+      - "https://docs.oracle.com/en/cloud/paas/autonomous-database/dedicated/adbde/"
+
+  - id: FK-023
+    area: "GPU Bare Metal — Preemptible Availability"
+    severity: HIGH
+    description: |
+      OCI Preemptible instances (~50% off) are technically supported for BM shapes,
+      but for high-demand GPU bare metal shapes (BM.GPU.H100, BM.GPU.H200, BM.GPU.B200,
+      BM.GPU.B300) preemptible capacity is practically unavailable. These shapes are
+      sold almost entirely via reservations or Universal Credits commitments.
+      Additionally, preemptible instances can be reclaimed with only 30 seconds notice,
+      making them operationally unviable for multi-day or multi-week training runs.
+    recommendation: |
+      Do NOT model GPU BM cost savings based on preemptible pricing.
+      Use Universal Credits 1yr (~35% off) or 3yr (~55% off) for realistic estimates.
+      If the workload is short batch jobs tolerant to interruptions, ask your PDM
+      about spot/preemptible availability before committing to that model.
+    source: "Field experience — Diego, OCI SA"

kb/field-knowledge/lessons-learned.yaml

@@ -0,0 +1,96 @@
+---
+last_verified: 2026-03-14
+description: |
+  Lessons learned from real OCI deployments. Anonymized but based on
+  actual customer experiences.
+---
+
+lessons:
+  - id: LL-001
+    category: migration
+    title: "Always test GoldenGate replication before cutover"
+    context: |
+      Customer planned zero-downtime migration using GoldenGate replication
+      from on-prem Exadata to ADB-S. During cutover rehearsal, discovered
+      that several PL/SQL packages used database links that didn't work with
+      ADB-S private endpoints.
+    lesson: |
+      Run full regression testing with GoldenGate replication active for
+      at least 2 weeks before cutover. Test all database links, PL/SQL
+      packages, and application connectivity against the target.
+    recommendation: "Add 2-week parallel-run phase to every migration plan"
+
+  - id: LL-002
+    category: sizing
+    title: "P95 is not enough for seasonal workloads"
+    context: |
+      Retail customer sized ADB-S for P95 utilization from normal months.
+      During Black Friday, actual peak was 4x normal (well above P95).
+      Auto-scaling helped but wasn't fast enough for the initial surge.
+    lesson: |
+      For seasonal workloads, get metrics from the peak period specifically.
+      Generic P95 from normal months underestimates seasonal peaks.
+    recommendation: "Ask for peak-period metrics separately, size auto-scale max for peak"
+
+  - id: LL-003
+    category: networking
+    title: "Plan IP addressing before anything else"
+    context: |
+      Customer deployed dev, staging, and production with overlapping CIDR
+      ranges. When they needed VCN peering for shared services, had to
+      re-architect networking — required rebuilding all subnets.
+    lesson: |
+      IP addressing must be planned enterprise-wide before deploying
+      any VCN. Include on-prem ranges, future VCNs, and DR regions.
+    recommendation: "Create IP addressing document as first deliverable"
+
+  - id: LL-004
+    category: operations
+    title: "IaC from day one, not after"
+    context: |
+      Customer built initial infrastructure manually via console, planning
+      to "Terraformify" later. After 6 months, infrastructure had drifted
+      so far that reverse-engineering Terraform was more work than starting over.
+    lesson: |
+      Start with Terraform from the first resource. The cost of IaC on day
+      one is low; the cost of retrofitting is very high.
+    recommendation: "Include OCI Terraform provider setup in project kickoff"
+
+  - id: LL-005
+    category: cost
+    title: "Auto-scaling costs can surprise"
+    context: |
+      Customer enabled ADB-S auto-scaling with 3x maximum. Workload was
+      consistently hitting auto-scale range, resulting in 2x the expected
+      database costs. They were effectively paying for auto-scaled OCPUs
+      most of the time.
+    lesson: |
+      If workload consistently auto-scales, the base OCPU is too low.
+      Right-size base to P75 and use auto-scaling for genuine peaks.
+    recommendation: "Review auto-scaling usage monthly, adjust base OCPUs accordingly"
+
+  - id: LL-006
+    category: dr
+    title: "DR drill found app-tier gap"
+    context: |
+      Customer had cross-region ADG for database DR. During DR drill,
+      database failed over successfully in 3 minutes. But app tier had
+      no DR — compute instances were only in primary region. Recovery
+      took 2 hours to provision app tier in DR region.
+    lesson: |
+      DR must cover all tiers, not just the database. App tier, load
+      balancers, DNS, and certificates all need DR strategy.
+    recommendation: "Include app tier DR in all architecture proposals"
+
+  - id: LL-007
+    category: security
+    title: "Cloud Guard catches what humans miss"
+    context: |
+      Customer deployed without Cloud Guard. 3 months later, security
+      audit found public Object Storage buckets, overly permissive
+      security lists, and an unused API key with admin privileges.
+      Cloud Guard would have flagged all of these on day one.
+    lesson: |
+      Enable Cloud Guard in every deployment. It's free and catches
+      common misconfigurations automatically.
+    recommendation: "Cloud Guard is mandatory in security baseline"

kb/field-knowledge/real-world-limits.yaml

@@ -0,0 +1,77 @@
+---
+last_verified: 2026-03-14
+description: |
+  Real-world limits that differ from official documentation or are not
+  well-documented. These come from actual deployment experience.
+---
+
+limits:
+  - service: ADB-S
+    limit: "Concurrent connections"
+    documented: "300 per OCPU"
+    real_world: |
+      300 per OCPU is the hard limit, but performance degrades noticeably
+      past 200 active sessions per OCPU. Size for 200 concurrent active
+      sessions per OCPU, not 300.
+    recommendation: "Plan for 200 active sessions per OCPU as practical limit"
+
+  - service: ADB-S
+    limit: "Auto-scaling speed"
+    documented: "Automatic"
+    real_world: |
+      Takes 2-3 minutes to begin scaling. Not instantaneous. Very short
+      bursts (< 3 minutes) may complete before scaling activates.
+    recommendation: "Size base OCPUs for P75 to absorb initial burst"
+
+  - service: ExaCS
+    limit: "Provisioning time"
+    documented: "Hours"
+    real_world: |
+      Can take 4-8 hours for infrastructure provisioning. VM cluster
+      creation adds another 1-2 hours. Total: 6-10 hours first time.
+    recommendation: "Plan provisioning 1-2 business days ahead"
+
+  - service: OKE
+    limit: "Node provisioning"
+    documented: "Minutes"
+    real_world: |
+      Node provisioning: 3-5 minutes for compute instance.
+      Node ready (kubelet + image pull): 5-10 minutes total.
+      Pod scheduling after node ready: 10-30 seconds.
+    recommendation: "Plan for 10-minute node scale-up, keep warm pool for critical workloads"
+
+  - service: FastConnect
+    limit: "Provisioning time"
+    documented: "Varies by partner"
+    real_world: |
+      Partner-based: 2-6 weeks depending on partner and location.
+      Co-location: 1-2 weeks if already in the same facility.
+      Always have IPSec VPN as backup during provisioning.
+    recommendation: "Start procurement 6-8 weeks early, use VPN as interim"
+
+  - service: Block Volume
+    limit: "IOPS per volume"
+    documented: "Up to 25K (Balanced), 35K (Higher Performance)"
+    real_world: |
+      Documented limits are achievable with sufficient volume size.
+      Small volumes (< 200 GB) won't reach max IOPS due to per-GB ratio.
+      Need 417 GB for 25K IOPS on Balanced (60 IOPS/GB).
+    recommendation: "Size volumes for IOPS requirement, not just capacity"
+
+  - service: Object Storage
+    limit: "API rate limits"
+    documented: "Request-based"
+    real_world: |
+      Burst throttling can occur at ~1000 requests/second per bucket.
+      Use multipart upload for large objects. Prefix partitioning helps
+      for high-throughput scenarios.
+    recommendation: "Use multipart upload, prefix partitioning for high throughput"
+
+  - service: ADB-S
+    limit: "Maximum databases per tenancy"
+    documented: "Service limits page"
+    real_world: |
+      Default limit is often 10-20 ADB instances per region. Need to
+      request limit increase via support ticket. Increases are usually
+      approved within 1-2 business days.
+    recommendation: "Request limit increases early in project planning"