226 lines
4.9 KiB
Markdown
226 lines
4.9 KiB
Markdown
# OCI DevOps Remediation Pipelines
|
|
|
|
This demo uses one detection path and three independent remediation pipelines. The preferred remediation style is declarative Kubernetes manifests applied by OCI DevOps.
|
|
|
|
## Detection flow
|
|
|
|
Use the main deploy or diagnosis pipeline to invoke the kagent Function with:
|
|
|
|
```text
|
|
k8s/kagent-multi-error-payload.json
|
|
```
|
|
|
|
kagent can return one or more problems:
|
|
|
|
```text
|
|
BUSINESS_PROCESS_DEGRADATION
|
|
IMAGE_PULL_FAILURE
|
|
HPA_AUTOSCALING_FAILURE
|
|
```
|
|
|
|
OCI DevOps does not provide a simple native OR/branch based on the Function JSON. Use one of these patterns:
|
|
|
|
```text
|
|
Manual demo:
|
|
Invoke Function -> Approval -> operator starts the correct remediation pipeline.
|
|
|
|
Automated demo:
|
|
Function parses kagent response and calls OCI DevOps API to start the correct remediation pipeline.
|
|
```
|
|
|
|
## Pipeline 1: Business application remediation
|
|
|
|
Pipeline name:
|
|
|
|
```text
|
|
kagent-remediate-business-config
|
|
```
|
|
|
|
Purpose:
|
|
|
|
```text
|
|
Fix order-service business degradation caused by ConfigMap thresholds or stuck review mode.
|
|
```
|
|
|
|
Artifacts:
|
|
|
|
```text
|
|
Name: remediate-business-configmap
|
|
Type: Kubernetes manifest
|
|
Path: k8s/remediation/business-configmap.yaml
|
|
|
|
Name: remediate-order-rollout-restart
|
|
Type: Kubernetes manifest
|
|
Path: k8s/remediation/order-rollout-restart.yaml
|
|
```
|
|
|
|
Recommended parameters:
|
|
|
|
```text
|
|
APP_NAMESPACE=kagent-demo
|
|
CONFIGMAP_NAME=demo-config
|
|
ORDER_MANUAL_REVIEW_RATE_THRESHOLD=70
|
|
ORDER_REVENUE_AT_RISK_THRESHOLD=15000
|
|
ORDER_STUCK_REVIEW_MODE=false
|
|
PAYMENT_FORCE_TIMEOUT=false
|
|
INVENTORY_FORCE_OUTAGE=false
|
|
WAIT_TIMEOUT=300s
|
|
```
|
|
|
|
Stages:
|
|
|
|
```text
|
|
approval-remediate-business
|
|
-> apply-business-configmap
|
|
-> apply-order-rollout-restart
|
|
-> wait-after-business-fix
|
|
-> invoke-kagent-function-validate
|
|
```
|
|
|
|
Important:
|
|
|
|
```text
|
|
order-service reads ConfigMap values as environment variables.
|
|
After applying business-configmap.yaml, apply order-rollout-restart.yaml with a new <restart-token>.
|
|
```
|
|
|
|
## Pipeline 2: ImagePull remediation
|
|
|
|
Pipeline name:
|
|
|
|
```text
|
|
kagent-remediate-imagepull-secret
|
|
```
|
|
|
|
Purpose:
|
|
|
|
```text
|
|
Restore the valid OCIR imagePullSecret in any business deployment: order-service, payment-service, or inventory-service.
|
|
```
|
|
|
|
Artifacts:
|
|
|
|
```text
|
|
Name: remediate-imagepull-order
|
|
Type: Kubernetes manifest
|
|
Path: k8s/remediation/imagepull-order.yaml
|
|
|
|
Name: remediate-imagepull-payment
|
|
Type: Kubernetes manifest
|
|
Path: k8s/remediation/imagepull-payment.yaml
|
|
|
|
Name: remediate-imagepull-inventory
|
|
Type: Kubernetes manifest
|
|
Path: k8s/remediation/imagepull-inventory.yaml
|
|
|
|
Name: remediate-imagepull-all-services
|
|
Type: Kubernetes manifest
|
|
Path: k8s/remediation/imagepull-all-services.yaml
|
|
```
|
|
|
|
Manifest placeholders:
|
|
|
|
```text
|
|
<region-key>
|
|
<tenancy-namespace>
|
|
<image-tag>
|
|
<restart-token>
|
|
```
|
|
|
|
Allowed values:
|
|
|
|
```text
|
|
DEPLOYMENT_NAME=order-service
|
|
DEPLOYMENT_NAME=payment-service
|
|
DEPLOYMENT_NAME=inventory-service
|
|
```
|
|
|
|
When remediating another service, set `CONTAINER_NAME` to the same value as `DEPLOYMENT_NAME`.
|
|
|
|
Stages:
|
|
|
|
```text
|
|
approval-remediate-imagepull
|
|
-> apply-imagepull-service-manifest
|
|
-> wait-after-imagepull-fix
|
|
-> invoke-kagent-function-validate
|
|
```
|
|
|
|
For exactly three remediation pipelines, use `imagepull-all-services.yaml`. It restores the imagePullSecret in all three services and avoids conditional branching.
|
|
|
|
Create one pipeline per target service only if you want narrower remediation:
|
|
|
|
```text
|
|
kagent-remediate-imagepull-order
|
|
kagent-remediate-imagepull-payment
|
|
kagent-remediate-imagepull-inventory
|
|
```
|
|
|
|
Or create one pipeline and manually select the matching manifest artifact based on kagent output.
|
|
|
|
## Pipeline 3: HPA capacity remediation
|
|
|
|
Pipeline name:
|
|
|
|
```text
|
|
kagent-remediate-order-hpa
|
|
```
|
|
|
|
Purpose:
|
|
|
|
```text
|
|
Increase HPA capacity when order-service reaches maxReplicas and demand still requires more pods.
|
|
```
|
|
|
|
Artifact:
|
|
|
|
```text
|
|
Name: remediate-order-hpa-capacity
|
|
Type: Kubernetes manifest
|
|
Path: k8s/remediation/order-hpa-capacity.yaml
|
|
```
|
|
|
|
Recommended parameters:
|
|
|
|
```text
|
|
APP_NAMESPACE=kagent-demo
|
|
HPA_NAME=order-service
|
|
MIN_REPLICAS=3
|
|
MAX_REPLICAS=8
|
|
CPU_TARGET=65
|
|
```
|
|
|
|
Stages:
|
|
|
|
```text
|
|
approval-remediate-hpa
|
|
-> apply-order-hpa-capacity
|
|
-> wait-after-hpa-fix
|
|
-> invoke-kagent-function-validate
|
|
```
|
|
|
|
## Demo execution order
|
|
|
|
Use this sequence for the conference:
|
|
|
|
```text
|
|
1. Run kagent-deployment-pipeline to deploy the healthy application.
|
|
2. Run kagent-simulate-failures-pipeline to inject controlled failures.
|
|
3. Run invoke-kagent-function with kagent-multi-error-payload.json.
|
|
4. Review kagent output.
|
|
5. Run one or more remediation pipelines:
|
|
- kagent-remediate-business-config
|
|
- kagent-remediate-imagepull-order/payment/inventory
|
|
- kagent-remediate-order-hpa
|
|
6. Run final kagent validation.
|
|
```
|
|
|
|
## Mapping from kagent response to remediation pipeline
|
|
|
|
```text
|
|
BUSINESS_PROCESS_DEGRADATION -> kagent-remediate-business-config
|
|
IMAGE_PULL_FAILURE -> kagent-remediate-imagepull-secret
|
|
HPA_AUTOSCALING_FAILURE -> kagent-remediate-order-hpa
|
|
NO_ACTIVE_FAILURE -> no remediation
|
|
```
|