AI gateway

An AI gateway is a service that sits between your app and many model providers and routes each call to the model you name. The gateway driver behind ai sends calls to a gateway instead of to one provider. Pick it when you want one key and one bill for models from several makers.

ai:
  driver: gateway

Before you start

1. Create an account with the gateway you use and an API key for your app, one per environment.
2. Turn on the providers and models you want to reach through it.
3. Check the gateway's data terms, and those of each provider behind it, against ai.data.

Configure

ai:
  driver: gateway
  key: secret:AI_GATEWAY_API_KEY     # suggested name; [to be decided] key field name for drivers that need one
  models: [anthropic/claude-*]       # allowlist, across every provider the gateway reaches
  data: { training: never, retention: zero }

Which gateways the driver supports, and how it is pointed at one, are not specified yet.

Use it

import { ai } from "mainsdk";

export async function summarise(text: string, user: { id: string }) {
  return ai.generate({
    model: "anthropic/claude-sonnet",   // [to be decided] model identifiers per driver
    prompt: `Summarise: ${text}`,
    feature: "summary",
    user,
  });
}

Local development and testing

Use ai.driver: local in main.env.dev.yaml to build without a key. createTestApp() from @mainsdk/testing runs ai on its in-memory fake, and chaos.on("ai", { outage: true }) simulates the gateway being down.

What MainSDK adds on top

• The key never in your code, only a secret: reference.
• Providers that train on or keep prompts are refused (MAIN-AI-0001 from mainsdk check).
• Personal data redacted before the call, so neither the gateway nor the provider sees it.
• Spend limits per user, per feature and in total, and every call audited without the prompt.

See AI calls.

Limits and notes

• The gateway driver comes with the ai spec (phase 1, milestone 0.3).
• A gateway adds a party that sees redacted prompts. Its data terms must meet ai.data as well as the provider's [ how that is declared].
• ai.raw() returns the gateway client; mainsdk check lists every use.

Swap to or from it

ai:
  driver: gateway                    # was bedrock

AI calls are stateless, so mainsdk swap tests the new driver and switches with no migration.

FAQ

Do I still need ai.models with a gateway? Yes. The allowlist applies whichever driver routes the call.

Do caps use the gateway's prices? Caps are enforced in your app before the call; how the gateway driver estimates cost is not decided yet.

Next

• AI calls
• Amazon Bedrock, Anthropic and OpenAI

Open questions