Alerts

Route SLO burn, downed dependencies, security detections and your own alerts to email, Slack, PagerDuty, Opsgenie or a webhook, once.

alerts:
  min: warning
  recipients:
    oncall:
      - { driver: pagerduty, key: secret:PAGERDUTY_ROUTING_KEY }
      - { driver: slack, webhook: secret:SLACK_ALERTS_WEBHOOK }
  routes:
    - { match: { severity: critical }, to: [oncall], escalate: [{ after: 15m, to: [oncall] }] }
    - { to: [oncall] }
  quiet: { from: "22:00", to: "08:00", tz: Asia/Kolkata }

import { alerts } from "mainsdk";

// one message per open key, however often it is raised; routes in main.yaml pick who hears it
await alerts.raise({ key: "import-failed", severity: "warning", title: "Nightly import failed", runbook: "https://acme.dev/runbooks/import" });

await alerts.resolve("import-failed");

import assert from "node:assert/strict";
import { createTestApp } from "@mainsdk/testing";

const app = createTestApp({
  yaml: `schema: 1
project: prj_alerts
alerts:
  recipients:
    oncall:
      - { driver: slack, webhook: secret:SLACK }
  routes:
    - { to: [oncall] }
`,
  secrets: { SLACK: "https://hooks.slack.com/services/T0/B0/xyz" },
});

await app.alerts.raise({ key: "import-failed", severity: "warning", title: "Import failed" });
await app.alerts.raise({ key: "import-failed", severity: "warning", title: "Import failed" });

const open = await app.alerts.list({ state: "open" });
assert.equal(open.items.length, 1);   // the second raise folded into the first
assert.equal(open.items[0]?.count, 2);

alerts:
  driver: builtin

Quick start

alerts:
  min: warning
  recipients:
    oncall:
      - { driver: pagerduty, key: secret:PAGERDUTY_ROUTING_KEY }
      - { driver: slack, webhook: secret:SLACK_ALERTS_WEBHOOK }
  routes:
    - { match: { severity: critical }, to: [oncall], escalate: [{ after: 15m, to: [oncall] }] }
    - { to: [oncall] }
  quiet: { from: "22:00", to: "08:00", tz: Asia/Kolkata }

import { alerts } from "mainsdk";

// one message per open key, however often it is raised; routes in main.yaml pick who hears it
await alerts.raise({ key: "import-failed", severity: "warning", title: "Nightly import failed", runbook: "https://acme.dev/runbooks/import" });

await alerts.resolve("import-failed");

Configuration

See the alerts.* rows in main.yaml reference. mainsdk check reports an invalid value as MAIN-ALERT-0001, a production app with no route for critical alerts as MAIN-ALERT-0002, and a literal key or a webhook host outside network.allow as MAIN-ALERT-0003.

What you get

• One message per open alert, however many instances raise it
• Critical alerts go at once; others wait out quiet hours as a digest
• Never on the request path; sends run as jobs with retries
• No personal data in an alert; provider dedupe keys are keyed hashes

Calls

Sources

The SDK raises alerts from these sources. Limit them with alerts.from.

The backup source (a failed or overdue restore drill) is planned. alerts.from accepts the name, but nothing raises from it yet.

Health and resource alerts resolve when the cause clears. Security, job and new-error alerts never resolve themselves; a person acknowledges them.

Recipients and destinations

A recipient is a named group of destinations in alerts.recipients. Each destination uses one of five drivers: email, slack, pagerduty, opsgenie or webhook. Keys and webhook URLs are written as secret:NAME, never as literals (MAIN-ALERT-0003). A route sends an alert to recipients; an escalation sends an open, unacknowledged alert to another recipient after a set time. Quiet hours hold warnings and info as one digest; critical alerts always go at once.

Drivers

CLI

mainsdk alerts test oncall --env prod
mainsdk info

alerts test sends a test alert, marked as a test, through every driver of the recipient and prints each result. mainsdk info shows the recipients by name and driver (never their keys), the routes and the open alerts.

Errors and checks

alerts.raise and alerts.resolve never wait on a provider. Sends run as jobs with retries. If a recipient fails after its last retry, the others are still sent to (MAIN-ALERT-0011). If the runtime store does not answer, dedupe and escalation fall back to the instance (MAIN-ALERT-0012) and the app keeps serving.

• MAIN-ALERT-0001
• MAIN-ALERT-0002
• MAIN-ALERT-0003
• MAIN-ALERT-0010
• MAIN-ALERT-0011
• MAIN-ALERT-0012
• MAIN-ALERT-0013
• MAIN-ALERT-0014
• MAIN-ALERT-0015

Each code's cause and fix is in check codes.

Swapping drivers

alerts has one driver, builtin, which sends through your notify and email drivers. To change where an alert goes, change the destination under a recipient (email, slack, pagerduty, opsgenie or webhook) and keep the calls as they are.

alerts:
  recipients:
    oncall:
      - { driver: opsgenie, key: secret:OPSGENIE_KEY }   # was pagerduty

See Swap a driver.

Testing

createTestApp runs alerts on the fake clock and captures outbound requests, so no provider is called. A raise with the key of an open alert folds into it.

import assert from "node:assert/strict";
import { createTestApp } from "@mainsdk/testing";

const app = createTestApp({
  yaml: `schema: 1
project: prj_alerts
alerts:
  recipients:
    oncall:
      - { driver: slack, webhook: secret:SLACK }
  routes:
    - { to: [oncall] }
`,
  secrets: { SLACK: "https://hooks.slack.com/services/T0/B0/xyz" },
});

await app.alerts.raise({ key: "import-failed", severity: "warning", title: "Import failed" });
await app.alerts.raise({ key: "import-failed", severity: "warning", title: "Import failed" });

const open = await app.alerts.list({ state: "open" });
assert.equal(open.items.length, 1);   // the second raise folded into the first
assert.equal(open.items[0]?.count, 2);

Next

• Raise and route alerts
• Alerts and health
• Notifications