> ## Documentation Index
> Fetch the complete documentation index at: https://docs.usekestrel.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Create Workflows
> Build automated workflows using natural language, drag-and-drop, CLI, SDK, or MCP
Kestrel workflows let you chain AI-powered actions across 15+ integrations into automated pipelines. Workflows are triggered by events — Kubernetes incidents, cloud alerts, Slack messages, PagerDuty alerts, and more — and execute a sequence of steps that investigate, fix, and notify.
## Creation Methods
You can create workflows five ways:
| Method | Best For |
| ------------------------ | ----------------------------------------------------------------------------------------------- |
| **Workflow Agent** | Fastest way to build — describe what you want in natural language, the Workflow Agent builds it |
| **Drag-and-Drop Canvas** | Visual construction with full control over step configuration |
| **CLI** | Scripting and automation — see [CLI reference](/workflows/cli) |
| **SDK** | Programmatic workflow management — see [SDK reference](/workflows/sdk) |
| **MCP** | AI agent orchestration (Cursor, Claude, etc.) — see [MCP reference](/workflows/mcp) |
***
## Workflow Agent
Describe the workflow you want in plain English. The Workflow Agent interprets your intent, selects the right triggers and actions, wires up template variables, and generates a complete workflow on the canvas.
Navigate to **Workflows → All Workflows** in the Kestrel dashboard and click **Create Workflow**.
Type a natural language description of what you want automated. Be specific about the trigger, the actions, and where results should go.
The Workflow Agent generates the workflow on the canvas. Review each step and iterate the prompt to adjust configuration, add, or remove steps as needed.
Click **Save** to activate the workflow. It will begin responding to its configured trigger immediately.
**Example prompts:**
* *"When a K8s incident is detected, run root cause analysis, send the RCA to #incidents in Slack, and create a Jira ticket with the details."*
* *"When someone requests a new service in Slack, generate the K8s manifests, open a GitOps PR in GitHub, and sync with ArgoCD."*
* *"When a Vercel deployment fails, pull the build logs, run an AI investigation, and notify the team in Slack."*
* *"When a Railway service crashes, fetch the logs, investigate the root cause, roll back to the previous deployment, and alert #platform in Slack."*
* *"When AWS costs spike, run a cost anomaly analysis and post a report to #finops in Slack."*
The more specific your description, the better the generated workflow. Include the trigger source, desired actions, and output destinations.
***
## Drag-and-Drop Canvas
The visual canvas gives you full control over workflow construction. Add triggers and actions from the sidebar, connect them into a pipeline, and configure each step individually.
### Adding a Trigger
Every workflow starts with a trigger. Drag a trigger block from the sidebar onto the canvas and configure it:
1. Select the trigger type (e.g., Pod CrashLoopBackOff, IAM Security Event, Slack Request)
2. Configure trigger-specific settings (clusters, namespaces, AWS accounts, severity thresholds)
3. The trigger's output variables become available to all downstream steps
### Adding Actions
Drag action blocks onto the canvas and connect them to the trigger or to previous actions:
1. Select the action type (e.g., Kestrel RCA, Slack Message, GitHub PR)
2. Configure action parameters — use the **variable picker** to reference data from previous steps
3. Connect the action's output port to the next step's input port
### Connecting Steps
Click an output port and drag to an input port to create a connection. Data flows top-to-bottom through the connections you define.
***
## Trigger Types
### Kubernetes Signals
Triggered when Kestrel detects a K8s incident in a connected cluster. Configure filters by cluster, namespace, or incident type.
| Trigger Block | Description |
| ---------------------------- | --------------------------------------------------------- |
| Deployment Replicas Failing | Triggers when a Deployment has unavailable replicas |
| Nodes Unavailable | Triggers when a node enters NotReady or unreachable state |
| Pod CrashLoopBackOff | Triggers when a pod enters CrashLoopBackOff |
| Pod ImagePullBackOff | Triggers when a pod cannot pull its container image |
| Pod OOMKilled | Triggers when a container is killed due to out-of-memory |
| Pods Failing | Triggers when a pod exits with a non-zero code |
| Pods Restarting | Triggers when pod restart count exceeds threshold |
| StatefulSet Replicas Failing | Triggers when a StatefulSet has unavailable replicas |
| Node Memory Pressure | Triggers when a node reports memory pressure |
| Node Disk Pressure | Triggers when a node reports disk pressure |
| DaemonSet Failing | Triggers when a DaemonSet has failing replicas |
| Any Kubernetes Incident | Triggers on any Kubernetes incident signal |
### AWS Cloud Signals
Triggered when Kestrel detects a cloud infrastructure incident. Filter by AWS account, region, or incident type.
| Trigger Block | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| IAM Security Event | Triggers on IAM compromise, privilege escalation, or unauthorized access |
| Root Account Activity | Triggers when the AWS root account is used |
| KMS Key Change | Triggers on KMS key deletion, disabling, or scheduled deletion |
| Secrets Manager Event | Triggers on secret deletion, policy changes, or rotation failures |
| Security Hub Finding | Triggers on high or critical findings from Security Hub |
| S3 Bucket Change | Triggers on S3 public access changes, policy modifications, or encryption changes |
| EC2 Instance Issue | Triggers on EC2 instance termination, state changes, or status check failures |
| Lambda Function Issue | Triggers on Lambda function errors, configuration issues, or throttling |
| ECS/EKS Container Issue | Triggers on container service task failures or deployment issues |
| RDS Database Issue | Triggers on RDS database deletion, snapshot issues, or availability problems |
| DynamoDB Issue | Triggers on DynamoDB table deletion, throttling, or capacity issues |
| VPC/Network Change | Triggers on VPC, security group, NACL, or route table modifications |
| CloudWatch Alarm | Triggers when a CloudWatch alarm enters ALARM state |
| Application Log Errors | Triggers when application errors are detected in CloudWatch Logs |
| Config Rule Non-Compliant | Triggers when an AWS Config rule finds a non-compliant resource |
| AWS Service Health Event | Triggers on AWS service health events affecting your resources |
| Any AWS Cloud Incident | Triggers on any AWS cloud incident |
| Cost Anomaly Detected | Triggers when AWS Cost Anomaly Detection finds an anomaly |
| Budget Threshold Exceeded | Triggers when an AWS Budget threshold is breached |
| Forecast Exceeds Budget | Triggers when the AWS cost forecast projects a budget overrun before it happens |
| Spend Spike Detected | Triggers when day-over-day spend rises beyond a configurable percentage threshold |
| Idle Resource Detected | Triggers when a daily scan finds idle resources (unattached EBS volumes, unassociated Elastic IPs, low-CPU instances, old snapshots) |
### Schedule
Triggered on a recurring cadence rather than by an external event. Ideal for periodic cost reports, cleanup scans, and audits. Configure the interval (hourly, daily, weekly, or monthly), the time of day (UTC), and — for weekly/monthly schedules — the day of week or day of month.
| Trigger Block | Description |
| ------------------ | -------------------------------------------------------------------- |
| Recurring Schedule | Triggers on a recurring schedule (hourly, daily, weekly, or monthly) |
Scheduled triggers expose `{{signal.schedule_interval}}`, `{{signal.fired_at}}`, and `{{signal.fired_date}}` to downstream steps.
### Slack Requests
Triggered when a user sends a `/kestrel-workflow` command in Slack. The user's message is available as `{{request.prompt}}`.
| Trigger Block | Description |
| ----------------------------------- | ----------------------------------------------------------------- |
| Create K8s Resource | Triggered when someone requests a new Kubernetes resource |
| Edit K8s Resource | Triggered when someone requests changes to a Kubernetes resource |
| General Kubernetes Request | Triggered when someone asks a Kubernetes-related question or task |
| Create AWS Resource | Triggered when someone requests a new AWS resource |
| Edit AWS Resource | Triggered when someone requests changes to an AWS resource |
| General Cloud Request | Triggered when someone asks a cloud-related question or task |
| Any Slack /kestrel-workflow Request | Triggers on any /kestrel-workflow request (catch-all) |
### PagerDuty
Triggered when a PagerDuty incident is created or updated. Filter by service, urgency, or severity.
| Trigger Block | Description |
| ---------------------- | -------------------------------------------------- |
| Incident Triggered | Triggers when a new PagerDuty incident is created |
| Incident Acknowledged | Triggers when a PagerDuty incident is acknowledged |
| Incident Resolved | Triggers when a PagerDuty incident is resolved |
| Any PagerDuty Incident | Triggers on any PagerDuty incident event |
| High Urgency Incident | Triggers only on high-urgency PagerDuty incidents |
### PostHog
Triggered by PostHog product analytics events. Filter by event type and properties.
| Trigger Block | Description |
| ----------------------- | ------------------------------------------------------ |
| Session Error/Exception | Triggers when PostHog captures a frontend exception |
| Console Error | Triggers when PostHog captures a console error |
| Rage Click | Triggers when PostHog detects rage clicks in a session |
| Any PostHog Event | Triggers on any PostHog event sent via webhook |
| Log Error Alert | Triggers when PostHog Logs detects error patterns |
### Vercel
Triggered by Vercel deployment and platform events. Filter by project and event type.
| Trigger Block | Description |
| ------------------------ | -------------------------------------------------------------- |
| Deployment Failed | Triggers when a Vercel deployment build fails |
| Deployment Succeeded | Triggers when a Vercel deployment succeeds |
| Deployment Created | Triggers when a new Vercel deployment is created |
| Error Anomaly | Triggers when Vercel detects a 5xx error rate anomaly |
| Usage Anomaly | Triggers when Vercel detects abnormal usage patterns |
| Domain Issue | Triggers when a Vercel project domain has configuration issues |
| Firewall Attack Detected | Triggers when Vercel's WAF detects an attack |
| Deployment Checks Failed | Triggers when post-deployment checks fail |
| Production Rollback | Triggers when a production deployment is rolled back |
### Railway
Triggered by Railway deployment and platform events. Filter by project, environment, and event type.
| Trigger Block | Description |
| --------------------- | ----------------------------------------------------------------- |
| Deployment Failed | Triggers when a Railway deployment build or deploy fails |
| Deployment Crashed | Triggers when a Railway service crashes after a successful deploy |
| Deployment Succeeded | Triggers when a Railway deployment succeeds and is live |
| Volume Usage Alert | Triggers when a Railway volume approaches its capacity |
| CPU/RAM Monitor Alert | Triggers when a Railway service exceeds CPU or memory thresholds |
### Fly.io
Triggered by Fly.io machine lifecycle changes. Fly.io has no outbound webhooks, so Kestrel detects these events by polling the Fly Machines API and diffing machine state on a configurable interval. Filter by app, region, and event type, and set the poll cadence (1m / 5m / 15m / 30m).
| Trigger Block | Description |
| --------------- | ------------------------------------------------------------ |
| Machine Crashed | Triggers when a Fly machine crashes (non-zero exit / OOM) |
| Machine Stopped | Triggers when a machine transitions to stopped/suspended |
| Machine Started | Triggers when a machine transitions to started |
| App Down | Triggers when all of an app's machines are stopped/unhealthy |
### Nebius AI Cloud
Triggered by Nebius GPU/compute and Managed Kubernetes node changes. Nebius has no outbound webhooks, so Kestrel detects these events by polling the Nebius compute and Managed Kubernetes APIs and diffing instance state and node conditions on a configurable interval. Filter by project, cluster, and event type, and set the poll cadence (1m / 5m / 15m / 30m).
| Trigger Block | Description |
| --------------------- | ------------------------------------------------------- |
| GPU Error | Triggers when a node reports a GPU error condition |
| Maintenance Scheduled | Triggers when a node has scheduled maintenance |
| Node Not Ready | Triggers when a Kubernetes node leaves the Ready state |
| Instance Stopped | Triggers when a compute instance transitions to stopped |
### Jenkins
Triggered by Jenkins build lifecycle events, delivered by a webhook notification plugin (or a post-build curl step) authenticated with a per-tenant shared secret. Filter by job name (full path for folder-nested jobs) and build status.
| Trigger Block | Description |
| ---------------------------- | ------------------------------------------------------------- |
| Build Failed | Triggers when a build completes with FAILURE |
| Build Unstable | Triggers when a build completes UNSTABLE (e.g. test failures) |
| Build Succeeded | Triggers when a build completes with SUCCESS |
| Build Completed (any result) | Triggers when a build finishes with any result |
| Build Started | Triggers when a build starts |
### CircleCI
Triggered by CircleCI's native outbound webhooks (workflow-completed / job-completed), signed with HMAC-SHA256 and verified by Kestrel. Filter by project slug, branch, and status.
| Trigger Block | Description |
| ------------------------------- | ---------------------------------------------------------------------------------------- |
| Workflow Failed | Triggers when a workflow completes with failed or error status |
| Workflow Succeeded | Triggers when a workflow completes successfully |
| Workflow Completed (any status) | Triggers when a workflow finishes with any status |
| Job Failed | Triggers when a job completes with failed status (finer-grained than workflow-completed) |
### Terraform Cloud
Triggered by Terraform Cloud (HCP Terraform) run lifecycle and drift events, delivered by per-workspace **notification configurations** (webhooks) signed with HMAC-SHA512 and verified by Kestrel. Filter by workspace, event type, and run status.
| Trigger Block | Description |
| ----------------------- | ---------------------------------------------------------------------------- |
| Run Created | Triggers when a new run is queued on a workspace |
| Run Planning | Triggers when a run starts planning |
| Run Needs Attention | Triggers when a plan finishes and the run awaits confirmation |
| Run Applying | Triggers when a run starts applying |
| Run Completed | Triggers when a run applies successfully or finishes as planned-and-finished |
| Run Errored | Triggers when a run fails (plan/apply error, policy hard-fail) |
| Drift Detected | Triggers when a workspace health assessment detects infrastructure drift |
| Assessment Check Failed | Triggers when a workspace health assessment fails to run |
### Pulumi Cloud
Triggered by Pulumi Cloud stack, update, deployment, drift, and policy events, delivered by organization or stack **webhooks** signed with HMAC-SHA256 (`Pulumi-Webhook-Signature`) and verified by Kestrel. Filter by stack (project/stack reference), project, and event type.
| Trigger Block | Description |
| -------------------- | -------------------------------------------------------------- |
| Update Succeeded | Triggers when a stack update (`pulumi up`) succeeds |
| Update Failed | Triggers when a stack update, refresh, or destroy fails |
| Preview Failed | Triggers when a stack preview fails |
| Destroy Succeeded | Triggers when a stack destroy succeeds |
| Deployment Started | Triggers when a Pulumi Deployments run starts executing |
| Deployment Succeeded | Triggers when a Pulumi Deployments run succeeds |
| Deployment Failed | Triggers when a Pulumi Deployments run fails |
| Drift Detected | Triggers when a drift detection run finds infrastructure drift |
| Drift Run Failed | Triggers when a drift detection or remediation run fails |
| Policy Violation | Triggers when a CrossGuard policy violation is detected |
| Stack Created | Triggers when a new stack is created in the organization |
| Stack Deleted | Triggers when a stack is deleted from the organization |
### Custom Webhook
Triggered by an HTTP POST to a unique webhook URL. Accepts any JSON payload with optional HMAC signature verification. See [Custom Integrations](/workflows/custom-integrations) for details.
***
## Action Types
Actions are grouped by integration. Each action produces output variables that downstream steps can reference.
Trigger K8s RCA & Generate Fix, Apply YAML Fix, Find Causal PRs/MRs, Trigger Cloud RCA & Generate Fix, Generate Runbook, Wait, Generate K8s Manifest, Apply K8s Manifest, Create GitOps PR, Generate Helm Values, Generate Cloud Resource, Execute Cloud CLI, Create IaC PR, Investigate Cloud, Investigate Kubernetes, AI Cost Analysis
Create Pull Request, Create Issue, Trigger GitHub Action, Wait for GitHub Action Run, Get GitHub Action Status, Read File, Search Code, AI Code Investigation, AI Code Fix, Wait for PR Approval, Wait for PR Merge
Create Merge Request, Create Issue, Trigger Pipeline, Wait for Pipeline, Get Pipeline Status, Wait for MR Approval, Wait for MR Merge
Send Message, Update Message, Request Justification
Publish RCA, Publish Postmortem, Publish Runbook Entry, Update Page
Create Ticket, Add Comment, Transition Ticket
Create Issue, Add Comment, Update Issue, Search Issues
Create Alert, Acknowledge Alert, Add Note to Alert, Resolve Alert, Escalate
Trigger ArgoCD Sync, Wait for ArgoCD Sync, Get ArgoCD App Status
Trigger Build, Wait for Build, Get Build Status, Stop Build, Get Console Log, Investigate Jenkins
Trigger Pipeline, Wait for Pipeline, Get Workflow Status, Rerun Workflow, Cancel Workflow, Approve On-Hold Job, Get Job Test Results, Investigate CircleCI
Query Metrics, Create Monitor, Send Event, Mute Monitor
Query Cost Explorer, Get Cost Anomalies, Get Cost Forecast, Get Budget Status, Get Rightsizing Recommendations, Get Savings Plans Recommendations, Get Reservation Recommendations, Get Commitment Utilization, Compare Cost Periods, Find Idle Resources, Get Compute Optimizer Recommendations, Get Trusted Advisor Cost Checks, Stop EC2 Instances, Delete Unattached EBS Volumes, Release Elastic IPs, Delete Old Snapshots
Get Session Summary, Get Session Recording, Query Events, List Session Recordings, Get Error Issue
Get Deployment, Get Build Logs, Rollback Production, Promote to Production, List Deployments, Investigate Vercel
Get Deployment, Get Deployment Logs, Rollback, Redeploy, Restart, List Deployments, Set Variables, Investigate Railway
Restart Machine, Start Machine, Stop Machine, Suspend Machine, Cordon Machine, Uncordon Machine, Get Machine, Get Machine Events, List Machines, Set Secrets, Investigate Fly
Get Instance, Start Instance, Stop Instance, Restart Instance, List Instances, List Clusters, List Node Groups, Scale Node Group, Investigate Nebius
List/Get Workspace, Lock/Unlock/Force-Unlock Workspace, List/Get Run, Create Run (Plan), Create Destroy Run, Apply Run, Discard Run, Cancel Run, Wait for Run, Get State Outputs, List Variables, Set Variable, Get Drift Assessment, Investigate Terraform
List/Get Stack, List/Get Update, Run Deployment, Get/Wait for/Cancel Deployment, Pause/Resume Deployments, Get Stack Outputs, Get Drift Status, Set/Delete Stack Tag, Investigate Pulumi
Wait for Manual Approval, Wait for Slack Approval, Wait for PR/MR Approval, Refine RCA with Feedback
Create custom action blocks to call any API endpoint, and custom webhook triggers to start workflows from external systems. See [Custom Integrations](/workflows/custom-integrations).
The Cloud Cost remediation blocks (**Stop EC2 Instances**, **Delete Unattached EBS Volumes**, **Release Elastic IPs**, **Delete Old Snapshots**) mutate real infrastructure. Always place them behind an [Approval Gate](#approval-gates). Resources tagged `kestrel:protected` are always skipped, every remediation block is pinned to an explicit region, and safety caps limit how many resources a single run can touch.
***
## Template Variables
Template variables let you pass data between workflow steps. Use double curly braces to reference outputs from triggers and previous actions.
### Variable Syntax
```
{{incident.title}} — Data from the trigger event
{{rca_result.root_cause}} — Promoted field from RCA actions
{{step_outputs.action-1.summary}} — Output from a specific action step
{{request.prompt}} — User input (for Slack request triggers)
```
### Common Variables
| Variable | Source | Description |
| ----------------------------------- | -------------------------- | ----------------------------------------------------------------------------------------- |
| `{{incident.title}}` | K8s/Cloud trigger | Incident title |
| `{{incident.namespace}}` | K8s trigger | Kubernetes namespace |
| `{{incident.resource_name}}` | K8s/Cloud trigger | Affected resource |
| `{{rca_result.root_cause}}` | Kestrel RCA action | Root cause summary |
| `{{rca_result.recommended_fix}}` | Kestrel RCA action | Recommended remediation |
| `{{fix.description}}` | Kestrel RCA action | Generated fix description |
| `{{step_outputs.action-1.summary}}` | Any action step | Summary output of the step |
| `{{request.prompt}}` | Slack request trigger | The user's original message |
| `{{request.requester}}` | Slack request trigger | The requesting user |
| `{{signal.deployment_id}}` | Vercel trigger | Vercel deployment ID |
| `{{signal.deployment_id}}` | Railway trigger | Railway deployment ID |
| `{{signal.service_id}}` | Railway trigger | Railway service ID |
| `{{signal.environment_id}}` | Railway trigger | Railway environment ID |
| `{{signal.app_name}}` | Fly.io trigger | Fly app name |
| `{{signal.machine_id}}` | Fly.io trigger | Fly machine ID |
| `{{signal.region}}` | Fly.io trigger | Fly region (e.g. iad) |
| `{{signal.exit_code}}` | Fly.io trigger | Machine exit code (crash events) |
| `{{signal.project_id}}` | Nebius trigger | Nebius project ID |
| `{{signal.instance_id}}` | Nebius trigger | Nebius compute instance ID |
| `{{signal.cluster_id}}` | Nebius trigger | Nebius Managed Kubernetes cluster ID |
| `{{signal.node_group_id}}` | Nebius trigger | Nebius node group ID |
| `{{signal.node_name}}` | Nebius trigger | Kubernetes node name |
| `{{signal.condition}}` | Nebius trigger | Node condition that fired (e.g. NebiusGPUError) |
| `{{signal.job_name}}` | Jenkins / CircleCI trigger | Jenkins job full name, or CircleCI job name (job-completed events) |
| `{{signal.build_number}}` | Jenkins trigger | Number of the build |
| `{{signal.build_status}}` | Jenkins trigger | Build result (SUCCESS, FAILURE, UNSTABLE, ABORTED) |
| `{{signal.build_url}}` | Jenkins trigger | Link to the build |
| `{{signal.project_slug}}` | CircleCI trigger | Project slug (e.g. gh/org/repo) |
| `{{signal.pipeline_id}}` | CircleCI trigger | ID of the pipeline |
| `{{signal.workflow_id}}` | CircleCI trigger | ID of the workflow |
| `{{signal.workflow_name}}` | CircleCI trigger | Name of the workflow |
| `{{signal.job_number}}` | CircleCI trigger | Number of the job (job-completed events) |
| `{{signal.status}}` | CircleCI trigger | Workflow/job status (success, failed, error, canceled) |
| `{{signal.branch}}` | CircleCI trigger | VCS branch the pipeline ran on |
| `{{signal.session_id}}` | PostHog trigger | PostHog session ID |
| `{{signal.workspace}}` | Terraform Cloud trigger | Workspace name |
| `{{signal.run_id}}` | Terraform Cloud trigger | Run ID (run-...) |
| `{{signal.run_status}}` | Terraform Cloud trigger | Run status at notification time |
| `{{signal.run_url}}` | Terraform Cloud trigger | Link to the run in Terraform Cloud |
| `{{signal.organization}}` | Terraform Cloud trigger | Terraform Cloud organization |
| `{{signal.session_id}}` | PostHog trigger | PostHog session ID |
| `{{signal.stack}}` | Pulumi Cloud trigger | Stack reference (project/stack) |
| `{{signal.project}}` | Pulumi Cloud trigger | Pulumi project name |
| `{{signal.operation}}` | Pulumi Cloud trigger | Operation kind (update, preview, refresh, destroy, detect-drift, remediate-drift) |
| `{{signal.result}}` | Pulumi Cloud trigger | Operation result (succeeded, failed) |
| `{{signal.update_version}}` | Pulumi Cloud trigger | Stack update version number |
| `{{signal.deployment_id}}` | Pulumi Cloud trigger | Pulumi Deployments run ID (deployment events) |
| `{{signal.update_url}}` | Pulumi Cloud trigger | Link to the update in the Pulumi Cloud console |
| `{{signal.user}}` | Pulumi Cloud trigger | User who initiated the operation |
| `{{signal.aws_account_id}}` | AWS cost trigger | AWS account ID for cost anomaly, budget, forecast, spend spike, and idle resource signals |
| `{{signal.schedule_interval}}` | Schedule trigger | Configured cadence (hourly, daily, weekly, monthly) |
| `{{signal.fired_at}}` | Schedule trigger | UTC timestamp (RFC3339) when the schedule fired |
| `{{signal.fired_date}}` | Schedule trigger | UTC date (YYYY-MM-DD) when the schedule fired |
### Variable Picker
When configuring an action step, click the **Variables** button on any text field to open the variable picker. It shows all available variables from the trigger and preceding steps, grouped by source. The picker is DAG-aware — it only shows variables that will be available at runtime based on the step's position in the workflow.
The variable picker only shows variables that are available at the current step's position in the workflow. Move a step earlier or later to change which variables it can access.
***
## Cooldown Configuration
Cooldowns prevent a workflow from firing repeatedly for the same event. Configure the cooldown period per workflow.
| Preset | Duration | Default For |
| ----------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| No cooldown | 0 | — |
| 5 minutes | 5m | — |
| 10 minutes | 10m | Slack, PostHog, Vercel, Railway, Fly.io, Nebius, Jenkins, CircleCI, Terraform Cloud, Pulumi Cloud, custom webhook triggers |
| 30 minutes | 30m | — |
| 1 hour | 1h | — |
| 3 hours | 3h | — |
| 12 hours | 12h | — |
| 24 hours | 24h | K8s incidents, cloud incidents, PagerDuty alerts |
Cooldowns are scoped to the specific trigger event. For K8s incidents, the cooldown applies per workload. For Slack requests, it applies per request.
***
## Approval Gates
Insert approval gates between steps to require human sign-off before the workflow continues. Three approval methods are available:
### Wait for Manual Approval
The workflow pauses and shows a pending approval in the Kestrel dashboard. An authorized user clicks **Approve** or **Reject**.
### Wait for Slack Approval
The workflow posts an approval request to a configured Slack channel with **Approve** and **Reject** buttons. Any member of the channel can respond.
### Wait for PR/MR Approval
The workflow waits for the associated pull request or merge request to be approved or merged (webhook-driven). Merging the PR approves the step; closing it rejects it.
### Refine RCA with Feedback (human-in-the-loop)
A self-looping review gate for incident-response workflows. Place it after a **Trigger K8s RCA** or **Trigger Cloud RCA** step. It presents the root cause and proposed fix for review with three outcomes:
* **Approve** — continues on the approved branch (e.g. apply the fix, publish the runbook).
* **Reject** — continues on the rejected branch.
* **Request changes** — you provide free-text guidance ("the real cause is the readiness probe timeout, not the image tag"), and the RCA agent **re-runs with your feedback** and comes back for review again. Feedback accumulates across rounds, so each pass builds on the last.
The loop repeats until you approve or reject, capped by **Max Refine Rounds** (default 5). When the cap is hit, the workflow advances on the approved branch. Choose **Kestrel UI** or **Slack** as the review channel in the block config.
***
## Poll Until (Loops)
The **Poll Until** block (Flow Control) repeatedly runs a single catalog action at a fixed interval and evaluates an exit condition against each iteration's output. It's the building block for "wait for X to reach state Y" automation — waiting for a sandbox to stop, a deployment to become healthy, a pipeline to finish, or a cloud resource to reach a target state.
Drag **Poll Until** from the sidebar (next to Conditions) onto the canvas, then configure:
* **Action to Poll** — any catalog action (searchable dropdown). Its config fields appear below and behave exactly like the standalone action block, including dynamic dropdowns (e.g. Cluster → Namespace scoping) and template variables.
* **Output Field Path** — which output field of the polled action to test, chosen from the action's known outputs.
* **Operator + Value** — `equals`, `not_equals`, `contains`, `not_contains`, `exists`, `not_exists`. Where the output has known discrete values (states, statuses) or refers to live resources (cluster IDs), the Value field is a dropdown. You can select **multiple values**: `equals`/`contains` are met when the output matches ANY selected value; `not_equals`/`not_contains` are met only when it matches NONE.
* **Poll Interval** — how often to re-run the action (default 60s, minimum 30s).
* **Timeout** — how long to keep polling (default 60 minutes).
The block has two outgoing branches:
* **met** — the exit condition held on some iteration; downstream steps see the final iteration's output.
* **timeout** — the timeout elapsed before the condition was met.
Polling is durable. The loop's schedule is persisted server-side, so a server restart mid-interval resumes the countdown where it left off instead of restarting the interval — and in-flight loops survive redeployments.
***
## Generating Runbooks
The **Generate Runbook** action (Kestrel) distills a completed RCA and its fixes into a reusable, generalized runbook for that *class* of incident — symptom, diagnosis steps, remediation steps, verification, and rollback. It requires an upstream RCA step. The generated runbook is available to downstream steps as `{{runbook_html}}` / `{{runbook_markdown}}`, and **Publish Runbook Entry** (Confluence) consumes `runbook_html` automatically.
***
## Testing Workflows
Before relying on a workflow in production, test it with a simulated trigger:
1. Open the workflow on the canvas
2. Click **Test Workflow**
3. Kestrel generates a sample trigger payload based on the trigger type
4. Click **Run Test** to execute the workflow with the sample data
5. Inspect the output of each step in the execution detail view
Test executions are marked as test runs in the [Observability dashboard](/workflows/observability) and do not count toward cooldowns.
***
## Example Workflows
### K8s Incident → RCA → Slack → Jira
**Workflow Agent prompt:** *"When a Kubernetes incident is detected in any cluster, run root cause analysis, send the root cause and recommended fix to #incidents in Slack, and create a Jira ticket in the INFRA project with the full investigation summary."*
| Step | Type | Configuration |
| ------------ | ------------------------------ | ------------------------------------------------------------------------------------------- |
| **Trigger** | Any Kubernetes Incident | All clusters |
| **Action 1** | Trigger K8s RCA & Generate Fix | Include metrics and logs |
| **Action 2** | Slack Send Message | Channel: `#incidents`, message: `{{incident.title}}` — `{{rca_result.root_cause}}` |
| **Action 3** | Jira Create Ticket | Project: INFRA, summary: `{{incident.title}}`, body: `{{rca_result.investigation_summary}}` |
### K8s Incident → RCA → Refine with Feedback → Apply → Publish Runbook
A full human-in-the-loop loop: detect the incident, run RCA, draft a reusable runbook, let an engineer iteratively refine the analysis, then apply the fix and publish the runbook.
**Workflow Agent prompt:** *"When a Kubernetes pod crashes, run root cause analysis, generate a runbook, then let me review and refine the analysis before applying — when I approve, apply the YAML fix and publish the runbook to Confluence; if I reject, notify #incidents."*
| Step | Type | Configuration |
| -------------------------- | ---------------------------------- | -------------------------------------------------------- |
| **Trigger** | Any Kubernetes Incident | All clusters |
| **Action 1** | Trigger K8s RCA & Generate Fix | Include metrics and logs |
| **Action 2** | Generate Runbook | Audience: SRE, include rollback |
| **Approval** | Refine RCA with Feedback | Channel: Kestrel UI, Max Refine Rounds: 3 |
| **On Approved → Action 3** | Apply YAML Fix | Auto-detected from RCA |
| **On Approved → Action 4** | Publish Runbook Entry (Confluence) | Uses `{{runbook_html}}` |
| **On Rejected → Action 5** | Slack Send Message | Channel: `#incidents`, message: RCA rejected by reviewer |
On **Request changes**, the reviewer's free-text guidance re-runs the RCA agent and the gate re-opens — repeating until approve/reject or the round cap.
### Slack Request → K8s Manifest → GitOps PR → ArgoCD Sync
**Workflow Agent prompt:** *"When someone requests a new Kubernetes resource via /kestrel-workflow in Slack, generate the K8s manifests based on their request, create a pull request in acme/k8s-manifests on GitHub, wait for the PR to be merged, then trigger an ArgoCD sync."*
| Step | Type | Configuration |
| ------------ | ----------------------------------- | ---------------------------- |
| **Trigger** | Any Slack /kestrel-workflow Request | Catch-all |
| **Action 1** | Generate K8s Manifest | Prompt: `{{request.prompt}}` |
| **Action 2** | Create GitOps PR | Repo: `acme/k8s-manifests` |
| **Approval** | Wait for PR/MR Approval | Wait for PR merge |
| **Action 3** | Trigger ArgoCD Sync | Auto-detected from PR |
### Vercel Deployment Failure → Build Logs → Investigation → Slack
**Workflow Agent prompt:** *"When a Vercel deployment fails, fetch the build logs, run an AI investigation to identify the root cause of the build failure, and send the findings to #deploys in Slack."*
| Step | Type | Configuration |
| ------------ | ------------------------ | ------------------------------------------------------------------ |
| **Trigger** | Vercel Deployment Failed | All projects |
| **Action 1** | Get Build Logs | Deployment: `{{signal.deployment_id}}` |
| **Action 2** | Investigate Vercel | Query: analyze build logs from `{{step_outputs.action-1.summary}}` |
| **Action 3** | Slack Send Message | Channel: `#deploys`, message: `{{step_outputs.action-2.summary}}` |
### Railway Deployment Crash → Logs → Investigation → Rollback → Slack
**Workflow Agent prompt:** *"When a Railway service crashes, fetch the deployment logs, run an AI investigation to find the root cause, roll the service back to its previous deployment, and post a summary to #platform in Slack."*
| Step | Type | Configuration |
| ------------ | -------------------------- | -------------------------------------------------------------------------- |
| **Trigger** | Railway Deployment Crashed | All projects |
| **Action 1** | Get Deployment Logs | Deployment: `{{signal.deployment_id}}` |
| **Action 2** | Investigate Railway | Query: analyze crash logs from `{{step_outputs.action-1.summary}}` |
| **Action 3** | Rollback | Service: `{{signal.service_id}}`, Environment: `{{signal.environment_id}}` |
| **Action 4** | Slack Send Message | Channel: `#platform`, message: `{{step_outputs.action-2.summary}}` |
### Fly.io Machine Crash → Events → Investigation → Restart → Slack
**Workflow Agent prompt:** *"When a Fly.io machine crashes, pull its recent events, run an AI investigation to find the root cause, restart the machine, and post a summary to #platform in Slack."*
| Step | Type | Configuration |
| ------------ | ---------------------- | ------------------------------------------------------------------------------------------------- |
| **Trigger** | Fly.io Machine Crashed | All apps, poll every 1m |
| **Action 1** | Get Machine Events | App: `{{signal.app_name}}`, Machine: `{{signal.machine_id}}` |
| **Action 2** | Investigate Fly | Query: find why machine `{{signal.machine_id}}` crashed using `{{step_outputs.action-1.summary}}` |
| **Action 3** | Restart Machine | App: `{{signal.app_name}}`, Machine: `{{signal.machine_id}}` |
| **Action 4** | Slack Send Message | Channel: `#platform`, message: `{{step_outputs.action-2.summary}}` |
### Nebius GPU Error → Investigation → Scale Node Group → Slack
**Workflow Agent prompt:** *"When a Nebius GPU node reports an error, run an AI investigation to find the root cause, add a node to the affected node group, and post a summary to #ml-infra in Slack."*
| Step | Type | Configuration |
| ------------ | ------------------ | ----------------------------------------------------------------------------------- |
| **Trigger** | Nebius GPU Error | All projects, poll every 5m |
| **Action 1** | Investigate Nebius | Query: find why node `{{signal.node_name}}` reported a GPU error |
| **Action 2** | Scale Node Group | Cluster: `{{signal.cluster_id}}`, Node Group: `{{signal.node_group_id}}`, Size: `3` |
| **Action 3** | Slack Send Message | Channel: `#ml-infra`, message: `{{step_outputs.action-1.summary}}` |
### Jenkins Build Failure → Console Log → Investigation → Slack + Jira
**Workflow Agent prompt:** *"When a Jenkins build fails, fetch the console log, run an AI investigation to identify the root cause of the failure, post the findings to #ci in Slack, and create a Jira ticket in the INFRA project."*
| Step | Type | Configuration |
| ------------ | -------------------- | ----------------------------------------------------------------------- |
| **Trigger** | Jenkins Build Failed | All jobs |
| **Action 1** | Get Console Log | Job: `{{signal.job_name}}`, Build: `{{signal.build_number}}` |
| **Action 2** | Investigate Jenkins | Query: analyze the console log from `{{step_outputs.action-1.summary}}` |
| **Action 3** | Slack Send Message | Channel: `#ci`, message: `{{step_outputs.action-2.summary}}` |
| **Action 4** | Jira Create Ticket | Project: `INFRA`, description: `{{step_outputs.action-2.answer}}` |
### CircleCI Workflow Failure → Rerun From Failed → Wait → PagerDuty
**Workflow Agent prompt:** *"When a CircleCI workflow fails on main, rerun only the failed jobs, wait for the pipeline to finish, and page on-call via PagerDuty with an AI investigation summary if the rerun also fails."*
| Step | Type | Configuration |
| ------------ | ------------------------ | ----------------------------------------------------------------------------------- |
| **Trigger** | CircleCI Workflow Failed | Projects: all, branch: `main` |
| **Action 1** | Rerun Workflow | Workflow: `{{signal.workflow_id}}`, from failed jobs only |
| **Action 2** | Wait for Pipeline | Project: `{{signal.project_slug}}`, Pipeline: `{{signal.pipeline_id}}`, timeout 30m |
| **Action 3** | Investigate CircleCI | Query: find why `{{signal.workflow_name}}` keeps failing (only on failure branch) |
| **Action 4** | PagerDuty Create Alert | Summary: `{{step_outputs.action-3.summary}}` |
### PostHog Exception → Investigation → Slack
**Workflow Agent prompt:** *"When PostHog captures a frontend exception, generate an AI summary of the user's session including what they were doing before the error, and send the summary with the replay URL to #product-alerts in Slack."*
| Step | Type | Configuration |
| ------------ | ------------------------------- | ---------------------------------------------------------------------------------------- |
| **Trigger** | PostHog Session Error/Exception | All events |
| **Action 1** | Get Session Summary | Session: `{{signal.session_id}}` |
| **Action 2** | Slack Send Message | Channel: `#product-alerts`, message: `{{step_outputs.action-1.summary}}` with replay URL |
### Terraform Run Needs Attention → Plan Summary → Slack Approval → Apply
**Workflow Agent prompt:** *"When a Terraform Cloud run on a production workspace finishes planning and needs confirmation, fetch the planned changes, ask for approval in #infra-approvals, and apply the run if approved or discard it if rejected."*
| Step | Type | Configuration |
| --------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------- |
| **Trigger** | Terraform Run Needs Attention | Workspaces: `prod-*` |
| **Action 1** | Get Run | Workspace: `{{signal.workspace}}`, Run: `{{signal.run_id}}` |
| **Action 2** | Wait for Slack Approval | Channel: `#infra-approvals`, message includes `{{step_outputs.action-1.summary}}` and `{{signal.run_url}}` |
| **On Approved** | Apply Run | Workspace: `{{signal.workspace}}`, Run: `{{signal.run_id}}` |
| **On Rejected** | Discard Run | Workspace: `{{signal.workspace}}`, Run: `{{signal.run_id}}` |
### Pulumi Update Failure → Investigation → Slack Approval → Retry
**Workflow Agent prompt:** *"When a Pulumi Cloud update fails on a production stack, investigate the failure, post the findings to #infra-approvals in Slack and ask for approval, then retry the update if approved."*
| Step | Type | Configuration |
| --------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------- |
| **Trigger** | Pulumi Update Failed | Stacks: `prod-*` |
| **Action 1** | Investigate Pulumi | Query: why did the update fail?, Stack: `{{signal.stack}}` |
| **Action 2** | Wait for Slack Approval | Channel: `#infra-approvals`, message includes `{{step_outputs.action-1.summary}}` and `{{signal.update_url}}` |
| **On Approved** | Run Deployment | Stack: `{{signal.stack}}`, Operation: `update` |
| **Action 4** | Wait for Deployment | Stack: `{{signal.stack}}`, Deployment: `{{step_outputs.action-3.deployment_id}}` |
### Cloud Cost Anomaly → AI Analysis → Slack Report
**Workflow Agent prompt:** *"When an AWS cost anomaly is detected, query the cost breakdown for the affected account over the last 7 days, run an AI cost analysis to identify the root cause and recommend optimizations, and post the analysis to #finops in Slack."*
| Step | Type | Configuration |
| ------------ | --------------------- | ---------------------------------------------------------------- |
| **Trigger** | Cost Anomaly Detected | All AWS accounts |
| **Action 1** | Query Cost Explorer | Account: `{{signal.aws_account_id}}`, time range: 7 days |
| **Action 2** | AI Cost Analysis | Type: anomaly |
| **Action 3** | Slack Send Message | Channel: `#finops`, message: `{{step_outputs.action-2.summary}}` |
### Weekly Schedule → Cost Report → Savings Recommendations → Slack
**Workflow Agent prompt:** *"Every Monday at 9am UTC, query last week's AWS costs grouped by service, get savings plans recommendations, run an AI cost analysis, and post an executive summary to #finops in Slack."*
| Step | Type | Configuration |
| ------------ | --------------------------------- | -------------------------------------------------------------------------------- |
| **Trigger** | Recurring Schedule | Weekly, Monday, 09:00 UTC |
| **Action 1** | Query Cost Explorer | Time range: 7 days, group by: Service |
| **Action 2** | Get Savings Plans Recommendations | Term: 1 year, payment: no upfront |
| **Action 3** | AI Cost Analysis | Type: summary |
| **Action 4** | Slack Send Message | Channel: `#finops`, message: `{{step_outputs.action-3.formatted_slack_message}}` |
### Idle Resource Scan → Approval → Cleanup → Slack
**Workflow Agent prompt:** *"Every day at 8am UTC, scan for idle resources (unattached EBS volumes and unassociated Elastic IPs), ask for approval in #finops, and if approved delete the unattached volumes (snapshot first) and release the idle Elastic IPs."*
| Step | Type | Configuration |
| -------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Trigger** | Recurring Schedule | Daily, 08:00 UTC |
| **Action 1** | Find Idle Resources | Resource types: EBS volumes, Elastic IPs; all enabled regions |
| **Action 2** | Wait for Slack Approval | Channel: `#finops`, message includes `{{step_outputs.action-1.idle_count}}` idle resources, est. savings `{{step_outputs.action-1.estimated_monthly_savings}}` |
| **On Approved → Action 3** | Delete Unattached EBS Volumes | Volume IDs: `{{step_outputs.action-1.idle_volume_ids}}`, snapshot first |
| **On Approved → Action 4** | Release Elastic IPs | Allocation IDs: `{{step_outputs.action-1.idle_eip_allocation_ids}}` |
| **Action 5** | Slack Send Message | Channel: `#finops`, message: cleanup summary |
### Forecast Overrun → Compare Periods → Rightsizing → Slack
**Workflow Agent prompt:** *"When the AWS cost forecast projects a budget overrun, compare this month's spend to last month to find the biggest movers, get rightsizing recommendations, and post the analysis to #finops in Slack."*
| Step | Type | Configuration |
| ------------ | ------------------------------- | ---------------------------------------------------------------- |
| **Trigger** | Forecast Exceeds Budget | All AWS accounts |
| **Action 1** | Compare Cost Periods | This month vs last month, group by: Service |
| **Action 2** | Get Rightsizing Recommendations | All recommendation types |
| **Action 3** | AI Cost Analysis | Type: optimization |
| **Action 4** | Slack Send Message | Channel: `#finops`, message: `{{step_outputs.action-3.summary}}` |
***
## Next Steps
* [Custom Integrations](/workflows/custom-integrations) — Build custom HTTP action blocks and webhook triggers for services not covered by built-in integrations
* [Workflow Integrations Setup](/workflows/setup-integrations) — Connect the integrations your workflows need
* [Workflow Observability](/workflows/observability) — Monitor execution metrics and debug failed runs