206 lines
5.1 KiB
Markdown
206 lines
5.1 KiB
Markdown
# OCI DevOps + kagent + OCI Function Remediation Flow
|
|
|
|
This runbook connects an OCI DevOps deployment pipeline with kagent through an OCI Function.
|
|
|
|
## 1. Expose kagent MCP privately
|
|
|
|
Edit the reserved private IP in:
|
|
|
|
```text
|
|
k8s/kagent/kagent-controller-private-lb.yaml
|
|
```
|
|
|
|
Replace:
|
|
|
|
```text
|
|
<reserved-private-ip>
|
|
```
|
|
|
|
with the OCI reserved private IP that belongs to the LoadBalancer subnet.
|
|
|
|
Apply the private LoadBalancer service:
|
|
|
|
```bash
|
|
kubectl apply -f k8s/kagent/kagent-controller-private-lb.yaml
|
|
kubectl get svc kagent-controller-private-lb -n kagent
|
|
```
|
|
|
|
Save the private IP and build the MCP URL:
|
|
|
|
```text
|
|
http://<private-kagent-lb-ip>:8083/mcp
|
|
```
|
|
|
|
The OCI Function subnet must be able to reach this private IP on TCP `8083`.
|
|
|
|
If the Service was previously created without the reserved private IP annotation, delete and recreate it:
|
|
|
|
```bash
|
|
kubectl delete svc kagent-controller-private-lb -n kagent
|
|
kubectl apply -f k8s/kagent/kagent-controller-private-lb.yaml
|
|
```
|
|
|
|
## 2. Deploy the OCI Function
|
|
|
|
Function path:
|
|
|
|
```text
|
|
oci-functions/kagent-devops-remediator
|
|
```
|
|
|
|
Configure these Function variables:
|
|
|
|
```text
|
|
KAGENT_MCP_URL=http://<private-kagent-lb-ip>:8083/mcp
|
|
KAGENT_AGENT_NAME=k8s-agent
|
|
KAGENT_AGENT_NAMESPACE=kagent
|
|
APP_NAMESPACE=kagent-demo
|
|
ORDER_OBSERVABILITY_URL=http://<order-lb-ip>/api/observability
|
|
```
|
|
|
|
The Function calls kagent MCP tool `invoke_agent` and requests a strict JSON diagnosis.
|
|
|
|
## 3. Add Function invocation to OCI DevOps
|
|
|
|
In the deploy pipeline, add a stage after the OKE deployment:
|
|
|
|
```text
|
|
Deploy application to OKE
|
|
-> Wait 60 seconds
|
|
-> Invoke Function: kagent-devops-remediator
|
|
-> Approval: review kagent diagnosis
|
|
-> Shell: apply ConfigMap fix
|
|
-> Shell: validate business health
|
|
```
|
|
|
|
The Function returns:
|
|
|
|
```json
|
|
{
|
|
"status": "HEALTHY|DEGRADED|FAILED",
|
|
"failureType": "BUSINESS_PROCESS_DEGRADATION",
|
|
"recommendedAction": "APPLY_CONFIGMAP_FIX",
|
|
"recommendedConfigMapData": {
|
|
"ORDER_MANUAL_REVIEW_RATE_THRESHOLD": "70",
|
|
"ORDER_REVENUE_AT_RISK_THRESHOLD": "15000"
|
|
},
|
|
"deploymentsToRestart": ["order-service"]
|
|
}
|
|
```
|
|
|
|
## 4. Apply the ConfigMap remediation
|
|
|
|
Use this command spec as a Shell deploy artifact:
|
|
|
|
```text
|
|
k8s/fix-business-config-command-spec.yaml
|
|
```
|
|
|
|
It patches `demo-config` and restarts `order-service`.
|
|
|
|
## 5. Validate the application
|
|
|
|
Use this command spec after remediation:
|
|
|
|
```text
|
|
k8s/validate-business-health-command-spec.yaml
|
|
```
|
|
|
|
Before using it, replace:
|
|
|
|
```text
|
|
http://<order-lb-ip>/api/observability
|
|
```
|
|
|
|
with the current LoadBalancer URL.
|
|
|
|
## Recommended pipeline behavior
|
|
|
|
Use this decision model:
|
|
|
|
```text
|
|
CONTINUE -> no remediation
|
|
APPROVAL_REQUIRED -> pause for human review
|
|
APPLY_CONFIGMAP_FIX -> approval, then run fix-business-config-command-spec.yaml
|
|
ROLLBACK -> run rollback stage or redeploy previous image tag
|
|
REDEPLOY -> rerun deploy stage with corrected artifact
|
|
```
|
|
|
|
For the conference demo, keep approval before remediation. It makes the control boundary clear: kagent diagnoses, OCI DevOps executes.
|
|
|
|
## Multi-error demo scenario
|
|
|
|
This demo can simulate three simultaneous issues:
|
|
|
|
```text
|
|
1. Business degradation:
|
|
ConfigMap thresholds make order-service report degraded business health.
|
|
|
|
2. Image pull failure:
|
|
payment-service is patched with an invalid imagePullSecret.
|
|
|
|
3. HPA ceiling:
|
|
order-service HPA is capped at 3 replicas.
|
|
```
|
|
|
|
Artifacts:
|
|
|
|
```text
|
|
k8s/kagent-multi-error-payload.json
|
|
k8s/broken/simulate-three-errors-command-spec.yaml
|
|
k8s/remediate-three-errors-command-spec.yaml
|
|
k8s/fix-business-config-command-spec.yaml
|
|
k8s/fix-imagepull-secret-command-spec.yaml
|
|
k8s/fix-order-hpa-command-spec.yaml
|
|
```
|
|
|
|
Recommended OCI DevOps flow:
|
|
|
|
```text
|
|
Deploy app
|
|
-> Wait 60-90 seconds
|
|
-> Optional Shell: simulate-three-errors-command-spec.yaml
|
|
-> Invoke Function with k8s/kagent-multi-error-payload.json
|
|
-> Approval: review kagent output
|
|
-> Shell: remediate-three-errors-command-spec.yaml
|
|
-> Shell: validate-business-health-command-spec.yaml
|
|
-> Invoke Function again to validate no active failures
|
|
```
|
|
|
|
The generic Function payload asks kagent to return a list of problems:
|
|
|
|
```json
|
|
{
|
|
"status": "HEALTHY|DEGRADED|FAILED",
|
|
"summary": "",
|
|
"problems": [
|
|
{
|
|
"failureType": "BUSINESS_PROCESS_DEGRADATION|IMAGE_PULL_FAILURE|HPA_AUTOSCALING_FAILURE|NO_ACTIVE_FAILURE",
|
|
"rootCause": "",
|
|
"evidence": [],
|
|
"impact": "",
|
|
"recommendedAction": "CONTINUE|APPROVAL_REQUIRED|APPLY_CONFIGMAP_FIX|FIX_IMAGE_PULL_SECRET|PATCH_HPA|ROLLBACK|REDEPLOY",
|
|
"affectedResources": [],
|
|
"recommendedConfigMapData": {},
|
|
"recommendedHpaPatch": {},
|
|
"deploymentsToRestart": []
|
|
}
|
|
],
|
|
"overallRecommendedAction": "CONTINUE|APPROVAL_REQUIRED|APPLY_CONFIGMAP_FIX|FIX_IMAGE_PULL_SECRET|PATCH_HPA|ROLLBACK|REDEPLOY"
|
|
}
|
|
```
|
|
|
|
For a first demo, use one remediation stage:
|
|
|
|
```text
|
|
remediate-three-errors-command-spec.yaml
|
|
```
|
|
|
|
For a more advanced demo, split remediation by action:
|
|
|
|
```text
|
|
APPLY_CONFIGMAP_FIX -> fix-business-config-command-spec.yaml
|
|
FIX_IMAGE_PULL_SECRET -> fix-imagepull-secret-command-spec.yaml
|
|
PATCH_HPA -> fix-order-hpa-command-spec.yaml
|
|
```
|