Build a durable workflow
Write a flow that sleeps, waits for an event and asks a person to approve as one async function, and watch it resume after a crash without repeating a step.
Status Status is where the work is: Planned (in a milestone, not started), Building (tickets in progress), Preview (usable, may change), then the stability it ships at.
Stability The promise an interface ships with.
- Stable Follows semantic versioning; breaking changes need a new major, 12 months of overlap and a codemod.
- Beta Usable in production; may change before it becomes stable.
- Experimental May change or go away; needs an explicit opt-in in main.yaml.
You will write an onboarding flow as one async function: send a welcome, wait three days, wait for the user to activate, ask a person to approve credits, then grant them. Each step is journaled, so a crash, a restart or a deploy resumes the run where it was and never repeats a finished step. It takes about 25 minutes.
You need
- A MainSDK app with
main.yaml. Production needs adbsection; the journal lives in your database. - Your own functions for the side effects (load a user, send an email, grant credits). The examples below stand in for them.
1. Declare the journal
workflows:
driver: db # the journal lives in your database
retries: { max: 3, backoff: exponential, initial: 1s } # the default for each stepworkflows:
driver: local # in process, for development and testsmainsdk check --env prod fails with MAIN-WORKFLOW-0011 when workflows uses the local driver, because runs would be lost when an instance is replaced.
2. Write the workflow
A workflow is a plain async function with a name and a version. Each step runs once and its JSON result is journaled. sleep, waitFor and approve park the run and hold no worker while they wait.
import { approve, sleep, step, waitFor, workflow } from "mainsdk/workflows";
import { s } from "mainsdk/schema";
// Your own code. Steps return JSON, so return ids and small values.
async function loadUser(userId: string) { return { id: userId, email: `${userId}@pinecart.dev` }; }
async function sendWelcome(email: string) { console.log("welcome sent to", email); return { sent: true }; }
async function grantCredits(userId: string, amount: number) { console.log("granted", amount, "to", userId); return { granted: amount }; }
export const onboarding = workflow(
"onboarding",
{ version: 1, input: s.object({ userId: s.string() }) },
async ({ userId }) => {
const user = await step("load-user", () => loadUser(userId));
await step("welcome", () => sendWelcome(user.email));
await sleep("3d"); // parks the run; holds no worker
const activated = await waitFor("user.activated", { match: { userId }, timeout: "7d" });
if (!activated) return { activated: false };
const ok = await approve({ action: "credits.grant", risk: "write", spend: "$5", reason: "Welcome credits", timeout: "2d" });
if (ok.approved) await step("grant", () => grantCredits(userId, 5), { retries: { max: 2 } });
return { activated: true, granted: ok.approved };
},
);The function runs again from the top on every resume, and each finished step returns its journaled result. So keep everything that reads a clock, picks a random number or calls the network inside a step. Outside a step it fails mainsdk check with MAIN-WORKFLOW-0010. Step results and the run's input must be JSON and at most 256KB (MAIN-WORKFLOW-0004).
3. Start a run
import { workflows, type WorkflowDef } from "mainsdk/workflows";
declare const onboarding: WorkflowDef<{ userId: string }>; // exported from app/workflows/onboarding.ts
const { runId } = await workflows.start(onboarding, { userId: "u_1" }, { idempotencyKey: "onboarding:u_1" });start writes the run and returns its runId; no step runs first. Starting again with the same idempotencyKey returns the same run. Look at where it is:
import { workflows } from "mainsdk/workflows";
declare const runId: string; // from workflows.start()
const run = await workflows.get(runId);
console.log(run.state, run.currentStep, run.history.map((h) => `${h.step}:${h.kind}`));A moment after start you should see sleeping sleep [ 'load-user:step', 'welcome:step', 'sleep:sleep' ]. The history lists names, kinds, attempts and times, never step results or inputs. state is queued, running, sleeping, waiting, completed, failed, cancelled or stuck.
4. Wake it with an event and a person
The waitFor resumes when an event of that type, whose data contains every match field, is published:
import { events } from "mainsdk";
await events.publish("user.activated", { userId: "u_1" });Then approve puts a request in the approval queue of agents and parks the run. A person approves it; an agent cannot approve its own request.
import { agents } from "mainsdk";
declare const session: { user: { id: string } }; // the signed-in approver
const [pending] = agents.pendingApprovals(); // what is waiting, with its reason and spend
agents.approve(pending.id, { kind: "user", id: session.user.id });After the approval, grant runs and the run completes with { activated: true, granted: true }. If nobody answers in two days, approve returns { approved: false, reason: "timeout" } and your code decides what that means.
5. Cancel or list runs
import { workflows } from "mainsdk/workflows";
declare const runId: string; // from workflows.start()
const waiting = await workflows.list({ name: "onboarding", state: "waiting" });
await workflows.cancel(runId, { reason: "user closed the account" });cancel stops a run that has not finished and records an audit event. On a finished run it fails with MAIN-WORKFLOW-0008.
6. Change the steps safely
A run keeps the version it started on. To change the steps, give the new function the next version and keep the old one registered until its runs finish. New runs start on the highest registered version.
import { workflow } from "mainsdk/workflows";
export const onboardingV2 = workflow("onboarding", { version: 2 /* , input, and the new steps */ }, async () => {
/* the new steps */
});Edit the steps under a live version and the run stops as stuck with MAIN-WORKFLOW-0002; unregister a version that still has runs and they stop as stuck with MAIN-WORKFLOW-0003 until you register it again.
7. Test it
createTestApp runs workflows on a fake clock and an in-memory journal, so a three-day sleep takes no time.
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { createTestApp } from "@mainsdk/testing";
import type { WorkflowDef } from "mainsdk/workflows";
declare const onboarding: WorkflowDef<{ userId: string }>; // exported from app/workflows/onboarding.ts
describe("onboarding workflow", () => {
it("workflows AC1 runs each step once, sleeps, waits, asks for approval and finishes", async () => {
const app = createTestApp();
const { runId } = await app.workflows.start(onboarding, { userId: "u_1" }, { idempotencyKey: "onboarding:u_1" });
await app.clock.tick("1m");
assert.equal((await app.workflows.get(runId)).state, "sleeping");
await app.clock.tick("3d"); // the sleep ends; now it waits for the event
assert.equal((await app.workflows.get(runId)).state, "waiting");
await app.events.publish("user.activated", { userId: "u_1" });
await app.clock.tick("1m"); // now it waits for a person
const [pending] = app.agents.pendingApprovals();
assert.equal(pending?.call.action, "credits.grant");
app.agents.approve(pending!.id, { kind: "user", id: "usr_1" });
await app.clock.tick("1m");
const run = await app.workflows.get(runId);
assert.equal(run.state, "completed");
assert.deepEqual(run.output, { activated: true, granted: true });
assert.deepEqual(run.history.map((h) => h.kind), ["step", "step", "sleep", "waitFor", "approve", "step"]);
});
it("workflows AC9 times out the wait and finishes without the grant", async () => {
const app = createTestApp();
const { runId } = await app.workflows.start(onboarding, { userId: "u_2" });
await app.clock.tick("11d");
const run = await app.workflows.get(runId);
assert.equal(run.state, "completed");
assert.deepEqual(run.output, { activated: false });
});
});Verify
npx mainsdk check --env prod
node --test app/workflows/onboarding.test.tsA pass prints mainsdk check: passed (prod) and both tests pass. With workflows.driver: local in main.yaml the prod check fails with MAIN-WORKFLOW-0011.
If it fails
| Code | Cause | Fix |
|---|---|---|
MAIN-WORKFLOW-0010 | A clock, random or network call in the workflow function outside a step | Move it into a step, or pass the value in as the input |
MAIN-WORKFLOW-0011 | The local driver in production | Set workflows.driver: db in main.yaml |
MAIN-WORKFLOW-0001 | step, sleep, waitFor or approve called outside a workflow run | Call it inside the function you pass to workflow(), started with workflows.start() |
MAIN-WORKFLOW-0002 | The code reached a step the journal doesn't have; the run is stuck | Restore the old steps, or start new runs on a new version |
MAIN-WORKFLOW-0003 | No code is registered for the run's version; the run is stuck | Register that version |
MAIN-WORKFLOW-0004 | A step result or input is not JSON, or is over max-step-result | Return a smaller JSON value; store large data and return its id |
MAIN-WORKFLOW-0005 | A step failed its last retry; the run is failed | Fix the cause and start a new run |
MAIN-WORKFLOW-0009 | A duration is not <n><s|m|h|d>, or is over 365 days | Use a value such as 3d |
MAIN-WORKFLOW-0012 | Two definitions share a name and version | Give one a new version |
For agents
Goal: turn <multi-step process> into one durable workflow.
Touch: main.yaml (workflows), main.env.dev.yaml (workflows.driver: local), one workflow file, the code that calls workflows.start(), one test.
Do: put every side effect, clock read, random value and network call inside a step; keep step results to JSON ids and small values; start runs with an idempotencyKey; give changed steps a new version.
Do not: edit the steps of a version that has live runs, use Date.now() or fetch outside a step, set the local driver in main.yaml, or loosen a check.
Verify: npx mainsdk check --env prod, then node --test on the workflow's test file.
Stop and ask if: a step has no safe way to run twice, or an approval needs a person who is not named.Next
- Durable workflows: every call, key and error.
- Run work in the background: one side effect, with retries and a dead-letter queue.
- Agents: the approval queue behind
approve.
Was this page helpful?