# OCI DevOps + kagent + OCI Function Remediation Flow

This runbook connects an OCI DevOps deployment pipeline with kagent through an OCI Function.

## 1. Expose kagent MCP privately

Edit the reserved private IP in:

```text
k8s/kagent/kagent-controller-private-lb.yaml
```

Replace:

```text
<reserved-private-ip>
```

with the OCI reserved private IP that belongs to the LoadBalancer subnet.

Apply the private LoadBalancer service:

```bash
kubectl apply -f k8s/kagent/kagent-controller-private-lb.yaml
kubectl get svc kagent-controller-private-lb -n kagent
```

Save the private IP and build the MCP URL:

```text
http://<private-kagent-lb-ip>:8083/mcp
```

The OCI Function subnet must be able to reach this private IP on TCP `8083`.

If the Service was previously created without the reserved private IP annotation, delete and recreate it:

```bash
kubectl delete svc kagent-controller-private-lb -n kagent
kubectl apply -f k8s/kagent/kagent-controller-private-lb.yaml
```

## 2. Deploy the OCI Function

Function path:

```text
oci-functions/kagent-devops-remediator
```

Configure these Function variables:

```text
KAGENT_MCP_URL=http://<private-kagent-lb-ip>:8083/mcp
KAGENT_AGENT_NAME=k8s-agent
KAGENT_AGENT_NAMESPACE=kagent
APP_NAMESPACE=kagent-demo
ORDER_OBSERVABILITY_URL=http://<order-lb-ip>/api/observability
```

The Function calls kagent MCP tool `invoke_agent` and requests a strict JSON diagnosis.

## 3. Add Function invocation to OCI DevOps

In the deploy pipeline, add a stage after the OKE deployment:

```text
Deploy application to OKE
  -> Wait 60 seconds
  -> Invoke Function: kagent-devops-remediator
  -> Approval: review kagent diagnosis
  -> Shell: apply ConfigMap fix
  -> Shell: validate business health
```

The Function returns:

```json
{
  "status": "HEALTHY|DEGRADED|FAILED",
  "failureType": "BUSINESS_PROCESS_DEGRADATION",
  "recommendedAction": "APPLY_CONFIGMAP_FIX",
  "recommendedConfigMapData": {
    "ORDER_MANUAL_REVIEW_RATE_THRESHOLD": "70",
    "ORDER_REVENUE_AT_RISK_THRESHOLD": "15000"
  },
  "deploymentsToRestart": ["order-service"]
}
```

## 4. Apply the ConfigMap remediation

Use this command spec as a Shell deploy artifact:

```text
k8s/fix-business-config-command-spec.yaml
```

It patches `demo-config` and restarts `order-service`.

## 5. Validate the application

Use this command spec after remediation:

```text
k8s/validate-business-health-command-spec.yaml
```

Before using it, replace:

```text
http://<order-lb-ip>/api/observability
```

with the current LoadBalancer URL.

## Recommended pipeline behavior

Use this decision model:

```text
CONTINUE              -> no remediation
APPROVAL_REQUIRED    -> pause for human review
APPLY_CONFIGMAP_FIX  -> approval, then run fix-business-config-command-spec.yaml
ROLLBACK             -> run rollback stage or redeploy previous image tag
REDEPLOY             -> rerun deploy stage with corrected artifact
```

For the conference demo, keep approval before remediation. It makes the control boundary clear: kagent diagnoses, OCI DevOps executes.

## Multi-error demo scenario

This demo can simulate three simultaneous issues:

```text
1. Business degradation:
   ConfigMap thresholds make order-service report degraded business health.

2. Image pull failure:
   payment-service is patched with an invalid imagePullSecret.

3. HPA ceiling:
   order-service HPA is capped at 3 replicas.
```

Artifacts:

```text
k8s/kagent-multi-error-payload.json
k8s/broken/simulate-three-errors-command-spec.yaml
k8s/remediate-three-errors-command-spec.yaml
k8s/fix-business-config-command-spec.yaml
k8s/fix-imagepull-secret-command-spec.yaml
k8s/fix-order-hpa-command-spec.yaml
```

Recommended OCI DevOps flow:

```text
Deploy app
  -> Wait 60-90 seconds
  -> Optional Shell: simulate-three-errors-command-spec.yaml
  -> Invoke Function with k8s/kagent-multi-error-payload.json
  -> Approval: review kagent output
  -> Shell: remediate-three-errors-command-spec.yaml
  -> Shell: validate-business-health-command-spec.yaml
  -> Invoke Function again to validate no active failures
```

The generic Function payload asks kagent to return a list of problems:

```json
{
  "status": "HEALTHY|DEGRADED|FAILED",
  "summary": "",
  "problems": [
    {
      "failureType": "BUSINESS_PROCESS_DEGRADATION|IMAGE_PULL_FAILURE|HPA_AUTOSCALING_FAILURE|NO_ACTIVE_FAILURE",
      "rootCause": "",
      "evidence": [],
      "impact": "",
      "recommendedAction": "CONTINUE|APPROVAL_REQUIRED|APPLY_CONFIGMAP_FIX|FIX_IMAGE_PULL_SECRET|PATCH_HPA|ROLLBACK|REDEPLOY",
      "affectedResources": [],
      "recommendedConfigMapData": {},
      "recommendedHpaPatch": {},
      "deploymentsToRestart": []
    }
  ],
  "overallRecommendedAction": "CONTINUE|APPROVAL_REQUIRED|APPLY_CONFIGMAP_FIX|FIX_IMAGE_PULL_SECRET|PATCH_HPA|ROLLBACK|REDEPLOY"
}
```

For a first demo, use one remediation stage:

```text
remediate-three-errors-command-spec.yaml
```

For a more advanced demo, split remediation by action:

```text
APPLY_CONFIGMAP_FIX     -> fix-business-config-command-spec.yaml
FIX_IMAGE_PULL_SECRET   -> fix-imagepull-secret-command-spec.yaml
PATCH_HPA               -> fix-order-hpa-command-spec.yaml
```