adding files 2
This commit is contained in:
Oracle Public Cloud User
205
OCI_DEVOPS_KAGENT_FUNCTION_FLOW.md
Normal file
205
OCI_DEVOPS_KAGENT_FUNCTION_FLOW.md
Normal file
@@ -0,0 +1,205 @@
|
||||
# OCI DevOps + kagent + OCI Function Remediation Flow
|
||||
|
||||
This runbook connects an OCI DevOps deployment pipeline with kagent through an OCI Function.
|
||||
|
||||
## 1. Expose kagent MCP privately
|
||||
|
||||
Edit the reserved private IP in:
|
||||
|
||||
```text
|
||||
k8s/kagent/kagent-controller-private-lb.yaml
|
||||
```
|
||||
|
||||
Replace:
|
||||
|
||||
```text
|
||||
<reserved-private-ip>
|
||||
```
|
||||
|
||||
with the OCI reserved private IP that belongs to the LoadBalancer subnet.
|
||||
|
||||
Apply the private LoadBalancer service:
|
||||
|
||||
```bash
|
||||
kubectl apply -f k8s/kagent/kagent-controller-private-lb.yaml
|
||||
kubectl get svc kagent-controller-private-lb -n kagent
|
||||
```
|
||||
|
||||
Save the private IP and build the MCP URL:
|
||||
|
||||
```text
|
||||
http://<private-kagent-lb-ip>:8083/mcp
|
||||
```
|
||||
|
||||
The OCI Function subnet must be able to reach this private IP on TCP `8083`.
|
||||
|
||||
If the Service was previously created without the reserved private IP annotation, delete and recreate it:
|
||||
|
||||
```bash
|
||||
kubectl delete svc kagent-controller-private-lb -n kagent
|
||||
kubectl apply -f k8s/kagent/kagent-controller-private-lb.yaml
|
||||
```
|
||||
|
||||
## 2. Deploy the OCI Function
|
||||
|
||||
Function path:
|
||||
|
||||
```text
|
||||
oci-functions/kagent-devops-remediator
|
||||
```
|
||||
|
||||
Configure these Function variables:
|
||||
|
||||
```text
|
||||
KAGENT_MCP_URL=http://<private-kagent-lb-ip>:8083/mcp
|
||||
KAGENT_AGENT_NAME=k8s-agent
|
||||
KAGENT_AGENT_NAMESPACE=kagent
|
||||
APP_NAMESPACE=kagent-demo
|
||||
ORDER_OBSERVABILITY_URL=http://<order-lb-ip>/api/observability
|
||||
```
|
||||
|
||||
The Function calls kagent MCP tool `invoke_agent` and requests a strict JSON diagnosis.
|
||||
|
||||
## 3. Add Function invocation to OCI DevOps
|
||||
|
||||
In the deploy pipeline, add a stage after the OKE deployment:
|
||||
|
||||
```text
|
||||
Deploy application to OKE
|
||||
-> Wait 60 seconds
|
||||
-> Invoke Function: kagent-devops-remediator
|
||||
-> Approval: review kagent diagnosis
|
||||
-> Shell: apply ConfigMap fix
|
||||
-> Shell: validate business health
|
||||
```
|
||||
|
||||
The Function returns:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "HEALTHY|DEGRADED|FAILED",
|
||||
"failureType": "BUSINESS_PROCESS_DEGRADATION",
|
||||
"recommendedAction": "APPLY_CONFIGMAP_FIX",
|
||||
"recommendedConfigMapData": {
|
||||
"ORDER_MANUAL_REVIEW_RATE_THRESHOLD": "70",
|
||||
"ORDER_REVENUE_AT_RISK_THRESHOLD": "15000"
|
||||
},
|
||||
"deploymentsToRestart": ["order-service"]
|
||||
}
|
||||
```
|
||||
|
||||
## 4. Apply the ConfigMap remediation
|
||||
|
||||
Use this command spec as a Shell deploy artifact:
|
||||
|
||||
```text
|
||||
k8s/fix-business-config-command-spec.yaml
|
||||
```
|
||||
|
||||
It patches `demo-config` and restarts `order-service`.
|
||||
|
||||
## 5. Validate the application
|
||||
|
||||
Use this command spec after remediation:
|
||||
|
||||
```text
|
||||
k8s/validate-business-health-command-spec.yaml
|
||||
```
|
||||
|
||||
Before using it, replace:
|
||||
|
||||
```text
|
||||
http://<order-lb-ip>/api/observability
|
||||
```
|
||||
|
||||
with the current LoadBalancer URL.
|
||||
|
||||
## Recommended pipeline behavior
|
||||
|
||||
Use this decision model:
|
||||
|
||||
```text
|
||||
CONTINUE -> no remediation
|
||||
APPROVAL_REQUIRED -> pause for human review
|
||||
APPLY_CONFIGMAP_FIX -> approval, then run fix-business-config-command-spec.yaml
|
||||
ROLLBACK -> run rollback stage or redeploy previous image tag
|
||||
REDEPLOY -> rerun deploy stage with corrected artifact
|
||||
```
|
||||
|
||||
For the conference demo, keep approval before remediation. It makes the control boundary clear: kagent diagnoses, OCI DevOps executes.
|
||||
|
||||
## Multi-error demo scenario
|
||||
|
||||
This demo can simulate three simultaneous issues:
|
||||
|
||||
```text
|
||||
1. Business degradation:
|
||||
ConfigMap thresholds make order-service report degraded business health.
|
||||
|
||||
2. Image pull failure:
|
||||
payment-service is patched with an invalid imagePullSecret.
|
||||
|
||||
3. HPA ceiling:
|
||||
order-service HPA is capped at 3 replicas.
|
||||
```
|
||||
|
||||
Artifacts:
|
||||
|
||||
```text
|
||||
k8s/kagent-multi-error-payload.json
|
||||
k8s/broken/simulate-three-errors-command-spec.yaml
|
||||
k8s/remediate-three-errors-command-spec.yaml
|
||||
k8s/fix-business-config-command-spec.yaml
|
||||
k8s/fix-imagepull-secret-command-spec.yaml
|
||||
k8s/fix-order-hpa-command-spec.yaml
|
||||
```
|
||||
|
||||
Recommended OCI DevOps flow:
|
||||
|
||||
```text
|
||||
Deploy app
|
||||
-> Wait 60-90 seconds
|
||||
-> Optional Shell: simulate-three-errors-command-spec.yaml
|
||||
-> Invoke Function with k8s/kagent-multi-error-payload.json
|
||||
-> Approval: review kagent output
|
||||
-> Shell: remediate-three-errors-command-spec.yaml
|
||||
-> Shell: validate-business-health-command-spec.yaml
|
||||
-> Invoke Function again to validate no active failures
|
||||
```
|
||||
|
||||
The generic Function payload asks kagent to return a list of problems:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "HEALTHY|DEGRADED|FAILED",
|
||||
"summary": "",
|
||||
"problems": [
|
||||
{
|
||||
"failureType": "BUSINESS_PROCESS_DEGRADATION|IMAGE_PULL_FAILURE|HPA_AUTOSCALING_FAILURE|NO_ACTIVE_FAILURE",
|
||||
"rootCause": "",
|
||||
"evidence": [],
|
||||
"impact": "",
|
||||
"recommendedAction": "CONTINUE|APPROVAL_REQUIRED|APPLY_CONFIGMAP_FIX|FIX_IMAGE_PULL_SECRET|PATCH_HPA|ROLLBACK|REDEPLOY",
|
||||
"affectedResources": [],
|
||||
"recommendedConfigMapData": {},
|
||||
"recommendedHpaPatch": {},
|
||||
"deploymentsToRestart": []
|
||||
}
|
||||
],
|
||||
"overallRecommendedAction": "CONTINUE|APPROVAL_REQUIRED|APPLY_CONFIGMAP_FIX|FIX_IMAGE_PULL_SECRET|PATCH_HPA|ROLLBACK|REDEPLOY"
|
||||
}
|
||||
```
|
||||
|
||||
For a first demo, use one remediation stage:
|
||||
|
||||
```text
|
||||
remediate-three-errors-command-spec.yaml
|
||||
```
|
||||
|
||||
For a more advanced demo, split remediation by action:
|
||||
|
||||
```text
|
||||
APPLY_CONFIGMAP_FIX -> fix-business-config-command-spec.yaml
|
||||
FIX_IMAGE_PULL_SECRET -> fix-imagepull-secret-command-spec.yaml
|
||||
PATCH_HPA -> fix-order-hpa-command-spec.yaml
|
||||
```
|
||||
Reference in New Issue
Block a user