Playbooks

playbooks lets you write down in main.yaml what your app does when something goes wrong, with dry-run, approval and automatic rollback. Nothing it does loosens anything, and nothing outlives 24 hours.

Examples: tighten the sign-in limit during an attack, switch off a heavy feature under CPU pressure, plan a key rotation. A rule matches signals, plans its steps, waits for a person unless you said auto, applies them as a short-lived overlay on your config, and undoes them when the condition clears, the time runs out or things get worse. The same guard (see reaction safety) covers every way a run can start. It is free.

playbooks:
  use: [attack, pressure, key-rotation]
  approve-timeout: 15m                 # how long an approve step waits
  presets:
    attack: { limits: [sign-in], posture: high, notify: oncall }
    pressure: { flags: [recommendations] }
  rules:
    report-flood:                      # a rule of your own
      when: { signal: anomaly.detected, min: medium }
      run: approve                     # auto | approve | dry-run
      do:
        - limits.tighten: { policy: report-export, to: 10/hour }
posture:
  levels:
    high: [alerts.raise]               # steps added to the level below
guard:
  max-active: 5                        # runs active at once, 1 to 20
  max-per-hour: 20
  max-hops: 2
  quarantine: 1h                       # how long a blocked source stays held
  limit-floor: 10%
  tenant-quorum: 3
  block:
    max-ranges: 256                    # ranges a playbook may add in total
    never: [10.0.0.0/8]                # ranges never blocked

import { playbooks } from "mainsdk";

// what would attack.contain change right now? nothing is applied
const plan = await playbooks.plan("attack.contain");
console.log(plan.why, plan.steps.map((s) => s.step));

playbooks:
  driver: builtin

Quick start

limits:
  policies:
    sign-in: { max: 10, per: 1m }
signals: {}
playbooks:
  use: [attack, pressure, key-rotation]
  presets:
    attack: { limits: [sign-in], posture: high }
    pressure: { flags: [recommendations] }

import { playbooks } from "mainsdk";

// what would attack.contain change right now? nothing is applied
const plan = await playbooks.plan("attack.contain");
console.log(plan.why, plan.steps.map((s) => `${s.step}: ${s.result}`));

Configuration

playbooks.mode is on, dry-run (every rule only plans) or off. use turns on presets, and presets.<name> gives them parameters. rules.<name> is your own rule:

playbooks:
  rules:
    report-flood:
      when: { signal: anomaly.detected, subject: "metric:requests", min: medium, count: { at-least: 3, within: 10m } }
      run: approve                   # auto | approve (default) | dry-run
      do:
        - limits.tighten: { policy: report-export, to: 10/hour }
        - flags.disable: { flag: bulk-export }
      for: 30m                       # how long steps hold; at most 24h
      until: cleared                 # also roll back when the signal lapses
      cooldown: 10m
      rollback-if: { errors: 2x }    # compare the 5 minutes after with the 5 before

The presets are attack.contain (approve), pressure.slim (auto), key-rotation.stale, key-rotation.compromised (approve) and key-rotation.rotated (auto). A step whose interface is not built yet (bots.block, alerts.raise, secrets.rotate, incident.sign-out, workers.scale, and action steps) is planned as skipped and mainsdk check says so with MAIN-PLAYBOOK-0002.

Steps available today

import { action } from "mainsdk";
import { s } from "mainsdk/schema";

export const freeze = action({
  name: "catalog.freeze",
  intent: "Freeze the catalogue",
  risk: "write",
  reaction: true,
  undo: "catalog.unfreeze",
  input: s.object({}),
  run: async () => ({ frozen: true }),
});

Calls and command line

npx mainsdk playbooks list
npx mainsdk playbooks plan attack.contain --signal <id>
npx mainsdk playbooks runs --state waiting
npx mainsdk playbooks approve <run>      # asks the running dev app
npx mainsdk playbooks revert <run> --reason "false alarm"

approve, decline and revert talk to a running mainsdk dev app through .mainsdk/control/. In production an approval is a person with the right permission, on the agents queue.

The guard

Every run, from a rule, a neighbour or a person, passes one guard, in order: never-a-reaction steps (MAIN-GUARD-0002: payments, data deletion, loosening, deploys, running programs, messaging users), protected targets (MAIN-GUARD-0003), loops (MAIN-GUARD-0004), quarantine after a rollback as worse (MAIN-GUARD-0007), max-per-hour (MAIN-GUARD-0006), max-active (MAIN-GUARD-0003), people steps in auto (MAIN-GUARD-0001, turned into an approval) and one tenant's signals driving an app-wide step (MAIN-GUARD-0005). Tune it with guard: (max-active, max-per-hour, quarantine, max-hops, block, limit-floor, tenant-quorum).

What you get

• Records each run with a one-sentence why, the signals that caused it and every step's result, in the audit log.
• Applies steps as an overlay that wins over pushed config and main.yaml, and removes it on expiry, clearing, worse or revert. The overlay is shared across instances through the store when one is configured.
• Rolls back as worse when the error rate rises by at least one point over the 5 minutes before, with at least 100 requests on each side.
• Never touches the request path beyond one overlay lookup (under 0.1 microseconds added).

A single serverless app has no long-lived instance for instance-class steps and no sampling for pressure.slim.

Helpers for wiring and tests

Errors and checks

MAIN-PLAYBOOK-0001 and 0002 and 0010, MAIN-GUARD-0001 and 0002 and 0008 come from mainsdk check. MAIN-PLAYBOOK-0003 to 0009 and MAIN-GUARD-0003 to 0007 come at run time. See check codes. Related: signals, norms, limits.

Swapping drivers

playbooks has no driver of its own. The overlay is shared across instances through the store when one is configured, so changing the store driver changes where it lives and your rules stay the same.

Testing

Run rules on createTestApp() from @mainsdk/testing with the fake clock. playbooks.plan(rule) changes nothing, so use it to assert what a rule would do, and new Playbooks(opts) builds the executor for custom wiring. See Test your app.

Next

• Signals
• Norms
• Limits
• Ops server