Tell the right person once with alerts
Declare who gets paged and how, raise an alert from your code, fold repeats into one message, acknowledge and resolve it, and send a test.
Status Status is where the work is: Planned (in a milestone, not started), Building (tickets in progress), Preview (usable, may change), then the stability it ships at.
Stability The promise an interface ships with.
- Stable Follows semantic versioning; breaking changes need a new major, 12 months of overlap and a codemod.
- Beta Usable in production; may change before it becomes stable.
- Experimental May change or go away; needs an explicit opt-in in main.yaml.
You will route an alert to a Slack channel and a pager, raise one from code, see fifty repeats become one message, and acknowledge it. It takes about 20 minutes. The SDK's own sources (health, security, resources, jobs, errors) raise alerts through the same router, so one setup covers them.
You need
- A MainSDK app with a
main.yaml, run from the repository (nothing here is published to npm). - A Slack incoming webhook and, if you page people, a PagerDuty routing key. Neither goes in code.
- For acknowledging, a role that holds the
alerts.managepermission.
1. Declare recipients and routes
alerts:
from: [health, security, jobs, errors]
min: warning
recipients:
oncall:
- { driver: pagerduty, key: secret:PAGERDUTY_ROUTING_KEY }
- { driver: slack, webhook: secret:SLACK_ALERTS_WEBHOOK }
routes:
- { match: { severity: critical }, to: [oncall] }
- { to: [oncall] }
group: { by: [source], wait: 30s, interval: 5m }
access:
roles:
admin: [alerts.manage]Keys and webhook URLs are secret:NAME references, never literals (MAIN-ALERT-0003). Production needs a route for critical (MAIN-ALERT-0002). Alerts below min are logged and sent to nobody.
2. Raise and resolve from code
import { alerts } from "mainsdk";
export async function onImportFailed(file: string) {
await alerts.raise({
key: "import-failed",
severity: "warning",
title: "Import failed",
runbook: "https://wiki.acme.dev/runbooks/import",
attrs: { file },
});
}
export async function onImportRecovered() {
await alerts.resolve("import-failed");
}A raise with the key of an open alert folds into it: count goes up and nothing more is sent. Put no personal data in title or attrs; user ids become keyed hashes and an attribute outside telemetry@1 is dropped (MAIN-ALERT-0013).
3. Acknowledge
import { alerts } from "mainsdk";
export async function POST(_req: Request, ctx: { params: { id: string } }) {
await alerts.ack(ctx.params.id); // needs alerts.manage, else MAIN-ALERT-0014 (403)
const open = await alerts.list({ state: "open" });
return Response.json(open);
}An acknowledged alert stops repeating and escalating. Who acknowledged is in the audit log.
4. Test the recipient
npx mainsdk alerts test oncallIt sends one message, marked as a test, through every driver of oncall and prints each result. npx mainsdk info lists recipients by name and driver (never keys), the routes and the open alerts.
Verify
npx mainsdk checkExit 0 means the alerts section reads, every route names a declared recipient and production has a critical route.
If it fails
| Code | What it means | Fix |
|---|---|---|
MAIN-ALERT-0001 | An unknown driver or source, a route to an undeclared recipient, or a bad quiet-hours time | Fix the value the finding names |
MAIN-ALERT-0002 | Production has alerts but no route for critical | Add { match: { severity: critical }, to: [...] } |
MAIN-ALERT-0003 | A webhook host is not in network.allow, or a key is a literal | Allow the host; use secret:NAME |
MAIN-ALERT-0011 | A recipient failed after its retries; the others were still sent to | Fix the recipient's endpoint or key |
MAIN-ALERT-0014 | The caller lacks alerts.manage | Give the role the permission |
For agents
Goal: route this app's operational alerts to oncall and raise one from the import job.
Touch: main.yaml (alerts, access.roles) and the import job.
Do not: call Slack, PagerDuty or email directly for an alert; put a key or URL in code; put personal data in a title or attrs.
Verify: npx mainsdk check, then npx mainsdk alerts test oncall.
Stop and ask if: there is no secret for the recipient, or nobody holds alerts.manage.Next
Was this page helpful?