Auto Remediation
Auto Remediation executes predefined runbooks — restart a service, scale a deployment, clear a queue, roll back a deploy — either automatically for known issues or with one-click approval for sensitive ones.
Runbooks#
A runbook is a versioned, parameterised set of steps. Applicare ships common ones and you can author your own with shell, HTTP, Kubernetes, or cloud actions.
runbook: restart-unhealthy-pods
params: { namespace, selector }
steps:
- k8s.rollout.restart(namespace, selector)
- wait.healthy(timeout=180s)
- notify.slack(#sre, "restarted {{selector}}")Automatic vs. approval#
Bind a runbook to a root-cause pattern for hands-off recovery of well-understood failures, or require approval for anything with blast radius. Every run is captured with who/what/when and the before-and-after state.
Start conservative: begin with approval-gated runbooks, watch them for a few incidents, then promote the reliable ones to fully automatic.
Audit & rollback#
Each remediation is fully audited and most define an inverse action, so a step can be rolled back if the health check after it fails.