kagent-oci-devops-demo/OCI_DEVOPS_KAGENT_FUNCTION_FLOW.md

OCI DevOps + kagent + OCI Function Remediation Flow

This runbook connects an OCI DevOps deployment pipeline with kagent through an OCI Function.

1. Expose kagent MCP privately

Edit the reserved private IP in:

k8s/kagent/kagent-controller-private-lb.yaml

Replace:

<reserved-private-ip>

with the OCI reserved private IP that belongs to the LoadBalancer subnet.

Apply the private LoadBalancer service:

kubectl apply -f k8s/kagent/kagent-controller-private-lb.yaml
kubectl get svc kagent-controller-private-lb -n kagent

Save the private IP and build the MCP URL:

http://<private-kagent-lb-ip>:8083/mcp

The OCI Function subnet must be able to reach this private IP on TCP 8083.

If the Service was previously created without the reserved private IP annotation, delete and recreate it:

kubectl delete svc kagent-controller-private-lb -n kagent
kubectl apply -f k8s/kagent/kagent-controller-private-lb.yaml

2. Deploy the OCI Function

Function path:

oci-functions/kagent-devops-remediator

Configure these Function variables:

KAGENT_MCP_URL=http://<private-kagent-lb-ip>:8083/mcp
KAGENT_AGENT_NAME=k8s-agent
KAGENT_AGENT_NAMESPACE=kagent
APP_NAMESPACE=kagent-demo
ORDER_OBSERVABILITY_URL=http://<order-lb-ip>/api/observability

The Function calls kagent MCP tool invoke_agent and requests a strict JSON diagnosis.

3. Add Function invocation to OCI DevOps

In the deploy pipeline, add a stage after the OKE deployment:

Deploy application to OKE
  -> Wait 60 seconds
  -> Invoke Function: kagent-devops-remediator
  -> Approval: review kagent diagnosis
  -> Shell: apply ConfigMap fix
  -> Shell: validate business health

The Function returns:

{
  "status": "HEALTHY|DEGRADED|FAILED",
  "failureType": "BUSINESS_PROCESS_DEGRADATION",
  "recommendedAction": "APPLY_CONFIGMAP_FIX",
  "recommendedConfigMapData": {
    "ORDER_MANUAL_REVIEW_RATE_THRESHOLD": "70",
    "ORDER_REVENUE_AT_RISK_THRESHOLD": "15000"
  },
  "deploymentsToRestart": ["order-service"]
}

4. Apply the ConfigMap remediation

Use this command spec as a Shell deploy artifact:

k8s/fix-business-config-command-spec.yaml

It patches demo-config and restarts order-service.

5. Validate the application

Use this command spec after remediation:

k8s/validate-business-health-command-spec.yaml

Before using it, replace:

http://<order-lb-ip>/api/observability

with the current LoadBalancer URL.

Recommended pipeline behavior

Use this decision model:

CONTINUE              -> no remediation
APPROVAL_REQUIRED    -> pause for human review
APPLY_CONFIGMAP_FIX  -> approval, then run fix-business-config-command-spec.yaml
ROLLBACK             -> run rollback stage or redeploy previous image tag
REDEPLOY             -> rerun deploy stage with corrected artifact

For the conference demo, keep approval before remediation. It makes the control boundary clear: kagent diagnoses, OCI DevOps executes.

Multi-error demo scenario

This demo can simulate three simultaneous issues:

1. Business degradation:
   ConfigMap thresholds make order-service report degraded business health.

2. Image pull failure:
   payment-service is patched with an invalid imagePullSecret.

3. HPA ceiling:
   order-service HPA is capped at 3 replicas.

Artifacts:

k8s/kagent-multi-error-payload.json
k8s/broken/simulate-three-errors-command-spec.yaml
k8s/remediate-three-errors-command-spec.yaml
k8s/fix-business-config-command-spec.yaml
k8s/fix-imagepull-secret-command-spec.yaml
k8s/fix-order-hpa-command-spec.yaml

Recommended OCI DevOps flow:

Deploy app
  -> Wait 60-90 seconds
  -> Optional Shell: simulate-three-errors-command-spec.yaml
  -> Invoke Function with k8s/kagent-multi-error-payload.json
  -> Approval: review kagent output
  -> Shell: remediate-three-errors-command-spec.yaml
  -> Shell: validate-business-health-command-spec.yaml
  -> Invoke Function again to validate no active failures

The generic Function payload asks kagent to return a list of problems:

{
  "status": "HEALTHY|DEGRADED|FAILED",
  "summary": "",
  "problems": [
    {
      "failureType": "BUSINESS_PROCESS_DEGRADATION|IMAGE_PULL_FAILURE|HPA_AUTOSCALING_FAILURE|NO_ACTIVE_FAILURE",
      "rootCause": "",
      "evidence": [],
      "impact": "",
      "recommendedAction": "CONTINUE|APPROVAL_REQUIRED|APPLY_CONFIGMAP_FIX|FIX_IMAGE_PULL_SECRET|PATCH_HPA|ROLLBACK|REDEPLOY",
      "affectedResources": [],
      "recommendedConfigMapData": {},
      "recommendedHpaPatch": {},
      "deploymentsToRestart": []
    }
  ],
  "overallRecommendedAction": "CONTINUE|APPROVAL_REQUIRED|APPLY_CONFIGMAP_FIX|FIX_IMAGE_PULL_SECRET|PATCH_HPA|ROLLBACK|REDEPLOY"
}

For a first demo, use one remediation stage:

remediate-three-errors-command-spec.yaml

For a more advanced demo, split remediation by action:

APPLY_CONFIGMAP_FIX     -> fix-business-config-command-spec.yaml
FIX_IMAGE_PULL_SECRET   -> fix-imagepull-secret-command-spec.yaml
PATCH_HPA               -> fix-order-hpa-command-spec.yaml