Amazon SQS

Amazon Simple Queue Service (SQS) is a planned driver behind events and jobs. jobs.run() puts work on an SQS queue, and a long-running worker takes it off, with MainSDK's retries, dead letters and idempotency on top. Pick it when your app runs on AWS with its own workers and you want jobs to stay in your account.

jobs:
  driver: sqs

Before you start

1. Create a queue in the AWS region your app runs in. SQS offers standard queues (at-least-once, best-effort order) and FIFO queues (ordered, with deduplication). Which kinds the driver supports is not decided yet.
2. Create a dead-letter queue for jobs that exhaust their retries, if you use dead-letter: true. How the driver maps MainSDK's dead-letter queue onto SQS redrive is not decided yet.
3. Give the producer and the worker IAM roles through workload identity: the producer can send messages, the worker can receive and delete them. Avoid long-lived access keys.

Configure

jobs:
  driver: sqs              # [to be decided] queue URL, region and other driver settings
  dead-letter: true
idempotency: { store: db, ttl: 24h }

Use it

Produce a job from a route handler:

import { auth, jobs } from "mainsdk";

export async function POST(req: Request) {
  const session = await auth.require(req);
  const { month } = await req.json();
  await jobs.run("reports.export", { userId: session.user.id, month }, {
    idempotencyKey: `export:${session.user.id}:${month}`,
  });
  return new Response(null, { status: 202 });
}

Consume it in a Node worker with the worker adapter, which restores the producer's trace and user:

import { worker } from "mainsdk/worker";

export default worker(async (job, ctx) => {
  if (job.name === "reports.export") {   // [to be decided] job object fields
    // build and send the report
  }
});

Local development and testing

• No AWS account: use the in-process local queue in a development overlay. mainsdk check refuses it in production.
• Tests: createTestApp() from @mainsdk/testing runs jobs on an in-memory fake, clock.advance() plays out retries, and chaos.on("jobs", { outage: true }) simulates SQS being unreachable.

What MainSDK adds on top

• CloudEvents envelope with versioned schemas for events.
• Retries with backoff and a dead-letter queue.
• A duplicate idempotency key never runs the side effect twice, from MainSDK's own record, even when SQS delivers a message more than once.

See Events and background jobs.

Limits and notes

• Planned driver, not in the spec yet: the spec defers it to the driver kit.
• SQS has no cron schedules of its own, so how jobs.schedule() runs on this driver is not decided yet.
• A queue lives in one AWS region; pick one that fits where job inputs may be stored (US, EU or India).
• Delivery is at least once with idempotent effects.

Swap to or from it

jobs:
  driver: sqs   # was inngest

FAQ

Do I need a worker? Yes. SQS holds messages; something has to poll and run them. Use the worker adapter from mainsdk/worker.

Standard or FIFO? Not decided for this driver yet. MainSDK's idempotency record makes duplicates safe on either.

Next

• Events and background jobs
• Inngest, Trigger.dev and AWS Secrets Manager

Open questions