kagent-oci-devops-demo/OCI_DEVOPS_REMEDIATION_PIPELINES.md

OCI DevOps Remediation Pipelines

This demo uses one detection path and three independent remediation pipelines. The preferred remediation style is declarative Kubernetes manifests applied by OCI DevOps.

Detection flow

Use the main deploy or diagnosis pipeline to invoke the kagent Function with:

k8s/kagent-multi-error-payload.json

kagent can return one or more problems:

BUSINESS_PROCESS_DEGRADATION
IMAGE_PULL_FAILURE
HPA_AUTOSCALING_FAILURE

OCI DevOps does not provide a simple native OR/branch based on the Function JSON. Use one of these patterns:

Manual demo:
Invoke Function -> Approval -> operator starts the correct remediation pipeline.

Automated demo:
Function parses kagent response and calls OCI DevOps API to start the correct remediation pipeline.

Pipeline 1: Business application remediation

Pipeline name:

kagent-remediate-business-config

Purpose:

Fix order-service business degradation caused by ConfigMap thresholds or stuck review mode.

Artifacts:

Name: remediate-business-configmap
Type: Kubernetes manifest
Path: k8s/remediation/business-configmap.yaml

Name: remediate-order-rollout-restart
Type: Kubernetes manifest
Path: k8s/remediation/order-rollout-restart.yaml

Recommended parameters:

APP_NAMESPACE=kagent-demo
CONFIGMAP_NAME=demo-config
ORDER_MANUAL_REVIEW_RATE_THRESHOLD=70
ORDER_REVENUE_AT_RISK_THRESHOLD=15000
ORDER_STUCK_REVIEW_MODE=false
PAYMENT_FORCE_TIMEOUT=false
INVENTORY_FORCE_OUTAGE=false
WAIT_TIMEOUT=300s

Stages:

approval-remediate-business
  -> apply-business-configmap
  -> apply-order-rollout-restart
  -> wait-after-business-fix
  -> invoke-kagent-function-validate

Important:

order-service reads ConfigMap values as environment variables.
After applying business-configmap.yaml, apply order-rollout-restart.yaml with a new <restart-token>.

Pipeline 2: ImagePull remediation

Pipeline name:

kagent-remediate-imagepull-secret

Purpose:

Restore the valid OCIR imagePullSecret in any business deployment: order-service, payment-service, or inventory-service.

Artifacts:

Name: remediate-imagepull-order
Type: Kubernetes manifest
Path: k8s/remediation/imagepull-order.yaml

Name: remediate-imagepull-payment
Type: Kubernetes manifest
Path: k8s/remediation/imagepull-payment.yaml

Name: remediate-imagepull-inventory
Type: Kubernetes manifest
Path: k8s/remediation/imagepull-inventory.yaml

Name: remediate-imagepull-all-services
Type: Kubernetes manifest
Path: k8s/remediation/imagepull-all-services.yaml

Manifest placeholders:

<region-key>
<tenancy-namespace>
<image-tag>
<restart-token>

Allowed values:

DEPLOYMENT_NAME=order-service
DEPLOYMENT_NAME=payment-service
DEPLOYMENT_NAME=inventory-service

When remediating another service, set CONTAINER_NAME to the same value as DEPLOYMENT_NAME.

Stages:

approval-remediate-imagepull
  -> apply-imagepull-service-manifest
  -> wait-after-imagepull-fix
  -> invoke-kagent-function-validate

For exactly three remediation pipelines, use imagepull-all-services.yaml. It restores the imagePullSecret in all three services and avoids conditional branching.

Create one pipeline per target service only if you want narrower remediation:

kagent-remediate-imagepull-order
kagent-remediate-imagepull-payment
kagent-remediate-imagepull-inventory

Or create one pipeline and manually select the matching manifest artifact based on kagent output.

Pipeline 3: HPA capacity remediation

Pipeline name:

kagent-remediate-order-hpa

Purpose:

Increase HPA capacity when order-service reaches maxReplicas and demand still requires more pods.

Artifact:

Name: remediate-order-hpa-capacity
Type: Kubernetes manifest
Path: k8s/remediation/order-hpa-capacity.yaml

Recommended parameters:

APP_NAMESPACE=kagent-demo
HPA_NAME=order-service
MIN_REPLICAS=3
MAX_REPLICAS=8
CPU_TARGET=65

Stages:

approval-remediate-hpa
  -> apply-order-hpa-capacity
  -> wait-after-hpa-fix
  -> invoke-kagent-function-validate

Demo execution order

Use this sequence for the conference:

1. Run kagent-deployment-pipeline to deploy the healthy application.
2. Run kagent-simulate-failures-pipeline to inject controlled failures.
3. Run invoke-kagent-function with kagent-multi-error-payload.json.
4. Review kagent output.
5. Run one or more remediation pipelines:
   - kagent-remediate-business-config
   - kagent-remediate-imagepull-order/payment/inventory
   - kagent-remediate-order-hpa
6. Run final kagent validation.

Mapping from kagent response to remediation pipeline

BUSINESS_PROCESS_DEGRADATION -> kagent-remediate-business-config
IMAGE_PULL_FAILURE           -> kagent-remediate-imagepull-secret
HPA_AUTOSCALING_FAILURE      -> kagent-remediate-order-hpa
NO_ACTIVE_FAILURE            -> no remediation