Moderation

moderate

Take reports, review them in a queue, decide and take appeals with a statement of reasons, and hold uploads until a pre-screen clears them.

Building for 0.4ships as a experimental APIPhase 1 · Core, public betaSpec moderateFree in your app, Apache-2.0

Status Status is where the work is: Planned (in a milestone, not started), Building (tickets in progress), Preview (usable, may change), then the stability it ships at.

Stability The promise an interface ships with.

Stable Follows semantic versioning; breaking changes need a new major, 12 months of overlap and a codemod.
Beta Usable in production; may change before it becomes stable.
Experimental May change or go away; needs an explicit opt-in in main.yaml.

Stability levels

View as Markdown Open in Claude Edit

Updated 2026-10-28

moderate takes reports, runs a review queue, decides with a statement of reasons, takes appeals and holds uploads until a pre-screen clears them. Reach for it when users post content that others see.

At a glance

moderate:
  reasons: [spam, harassment, hate]
  terms: terms
  hold: { files: true, media: true, max: 24h }
  prescreen:
    classify: { labels: [ok, spam, sexual, violence, hate], hold-on: [sexual, violence, hate], remove-on: [spam] }
    hash: { driver: none }
  reports: { per-user: 20/hour }
  appeals: { window: 6mo }

import { moderate } from "mainsdk/experimental/moderate";

await moderate.report({ target: { kind: "post", id: post.id }, reason: "harassment", by: user });

Calls

moderate.report({ target, reason, by })
moderate.screen({ target, author, text?, files? })
moderate.state(target)
moderate.queue.list()
moderate.queue.assign(caseId, to)
moderate.decide(caseId, { action, ground, facts, by })
moderate.appeal(decisionId, { by, note })
moderate.resolve(appealId, { outcome, note, by })
moderate.statement(decisionId)
moderate.transparency({ from, to })
moderate.watch("decision", fn)
moderate.watch("match", fn)
sanitize(object: html, markdown, text, attr, url)

Guarantees

Whatever the driver.

Uploads and media outputs are held until the pre-screen clears them
No hash list ships in the repository; lists come from your provider at run time
Every decision has a ground and a statement of reasons, and can be appealed to a different reviewer
Reports are rate-limited and every step is in the audit log

Drivers

4 drivers. Select one to see its line in main.yaml.

main.yamlBuilt in on db (default)

moderate:
  driver: builtin

Change this one line to swap providers; your calls stay the same.Built-in and local drivers integration

Codes

Code	Where it appears	Meaning
`MAIN-MOD-0010`	Checkmainsdk check, before deploy	A hash list or hash database in the repository or the build output
`MAIN-MOD-0011`	Checkmainsdk check, before deploy	User content is published (`feeds.publish`, `share` of an upload, a public file URL) with no `moderate` section
`MAIN-MOD-0012`	Checkmainsdk check, before deploy	No `moderate.reports.per-user`, or one above 100 an hour
`MAIN-MOD-0013`	Checkmainsdk check, before deploy	`appeals` off or a window under 6 months, or `remove-on` lists a label other than `spam`
`MAIN-SANITIZE-0003`	Checkmainsdk check, before deploy	A raw HTML sink (`dangerouslySetInnerHTML`, `innerHTML`, `outerHTML`, `insertAdjacentHTML`, `document.write`, `v-html`, `{@html}`) is fed by a value that doesn't come from `sanitize.html` or `sanitize.markdown`
`MAIN-MOD-0001`	RuntimeError responses and logs	Report refused: the person passed `reports.per-user`, or already reported this target (429)
`MAIN-MOD-0002`	RuntimeError responses and logs	Unknown reason, or the target doesn't exist in the current tenant (400)
`MAIN-MOD-0003`	RuntimeError responses and logs	The caller lacks `moderate.review`, or the case is assigned to someone else (403)
`MAIN-MOD-0004`	RuntimeError responses and logs	Appeal refused: outside `appeals.window`, already appealed, or the decision isn't the caller's to appeal (409)
`MAIN-MOD-0005`	RuntimeError responses and logs	An appeal resolved by the reviewer who made the decision (403)
`MAIN-MOD-0006`	RuntimeError responses and logs	A pre-screen driver is unreachable or failed; the content stays held and is retried, then queued for people after `hold.max` (logged)
`MAIN-MOD-0007`	RuntimeError responses and logs	Known-abuse hash match: the content is blocked and never published; logged with the list name, never the content
`MAIN-MOD-0008`	RuntimeError responses and logs	A decision other than `none` without a ground or facts (400)
`MAIN-MOD-0009`	RuntimeError responses and logs	Held, blocked or removed content read through `files.url`, `media.url` or `feeds` (403)
`MAIN-SANITIZE-0001`	RuntimeError responses and logs	See the spec
`MAIN-SANITIZE-0002`	RuntimeError responses and logs	See the spec

Features

Nothing here has shipped yet. Each feature lands in its phase; the spec ticket builds it.

Feature	Phase	Milestone	Spec ticket
moderate@1 contract	Phase 1	0.4	`moderate#contract`
Reports and cases	Phase 1	0.4	`moderate#reports`
Review queue and assignment	Phase 1	0.4	`moderate#queue`
Decisions: remove, label, restrict, warn	Phase 1	0.4	`moderate#decisions`
Appeals	Phase 1	0.4	`moderate#appeals`
Hold before publish	Phase 1	0.4	`moderate#hold`
Hash matching hook	Phase 1	0.4	`moderate#hash-hook`
Pre-screen with ai.classify	Phase 1	0.4	`moderate#classify`
Statements of reasons and DSA export	Phase 1	0.4	`moderate#statements`
Transparency report	Phase 1	0.4	`moderate#transparency`
Audit	Phase 1	0.4	`moderate#audit-tenancy`
Moderation checks	Phase 1	0.4	`moderate#checks`

Quick start

main.yaml

moderate:
  hold: { files: true, media: true, max: 24h }
  reports: { per-user: 20/hour }
  appeals: { window: 6mo }

app/report.ts

import { moderate } from "mainsdk/experimental/moderate";

declare const post: { id: string };
declare const user: { id: string };

await moderate.report({ target: { kind: "post", id: post.id }, reason: "harassment", by: user });

Configuration

Key	Meaning
`reasons`	The reasons a report may give
`terms`	Where grounds of kind `terms` point
`hold`	Keep uploads (`files`) and media outputs (`media`) held until cleared, for at most `max` before they go to the human queue
`prescreen.classify`	Labels, which labels hold and open a case (`hold-on`), and which remove automatically (`remove-on`)
`prescreen.hash`	Known-abuse hash matching: `none` (default), `pdq` or `photodna`, with lists from your provider
`queue`	Assignment (`round-robin`) and the `overdue` time
`reports`	Limits: `per-user` and `per-target`
`appeals`	`window`, at least 6 months

What you get

Reports, a review queue, decisions with a ground and facts, appeals to a different reviewer, a statement of reasons for every decision and a transparency report. The calls:

A target is { kind, id }, such as a post, comment, file, media item or account.

Call	What it does
`moderate.report({ target, reason, by })`	File a report; returns the report id and its case id
`moderate.screen({ target, author, text?, files? })`	Pre-screen new content; returns `clear`, `held` or `blocked`
`moderate.state(target)`	Read a target's state, labels and any warning
`moderate.queue.list()`	List cases, filtered by state, reason or assignee. Needs `moderate.review`
`moderate.queue.assign(caseId, to)`	Assign a case to a reviewer, or to `next`
`moderate.decide(caseId, { action, ground, facts, by })`	Decide: `remove`, `label`, `restrict`, `warn` or `none`. Any action but `none` needs a ground and facts
`moderate.appeal(decisionId, { by, note })`	Appeal a decision inside the window
`moderate.resolve(appealId, { outcome, note, by })`	Resolve an appeal as `upheld` or `reversed`, by someone other than the original reviewer
`moderate.statement(decisionId)`	Return the statement of reasons for a decision
`moderate.transparency({ from, to })`	Return counts of reports, decisions and appeals for a period
`moderate.watch("decision", fn)`	Run `fn` on each decision; returns an unsubscribe function
`moderate.watch("match", fn)`	Run `fn` on a known-abuse hash match; returns an unsubscribe function

Sanitizing user content

sanitize is built and ships with the SDK. Use it on the text users post before you show it.

Call	What it does
`sanitize(object: html, markdown, text, attr, url)`	The `sanitize` object has five members. `sanitize.html` keeps allowlisted markup, `sanitize.markdown` renders Markdown with raw HTML escaped, `sanitize.text` and `sanitize.attr` escape for a text node and a quoted attribute, and `sanitize.url` returns a cleaned URL or `null` when it is unsafe

Errors and checks

Code	When
`MAIN-MOD-0001`	A report is refused: over `reports.per-user`, or this target is already reported
`MAIN-MOD-0002`	Unknown reason, or the target is not in this tenant
`MAIN-MOD-0003`	The caller lacks `moderate.review`, or the case is someone else's
`MAIN-MOD-0004`	An appeal is outside the window, repeated, or not the caller's
`MAIN-MOD-0005`	An appeal is resolved by the reviewer who decided
`MAIN-MOD-0008`	A decision other than `none` has no ground or facts
`MAIN-MOD-0009`	Held, blocked or removed content is read through `files.url`, `media.url` or `feeds`

MAIN-MOD-0006 and 0007 are logged, not thrown. The check codes MAIN-MOD-0010 to 0013 find a hash list in the repository, user content published without moderation, missing report limits and appeals that are off or too short. See check codes and errors.

Swapping drivers

Driver	Provider
`builtin`	Built in on the database	default
`ai-classify`	Pre-screen through `ai.classify`
`pdq`	PDQ hash matching, lists from your provider
`photodna`	PhotoDNA, lists from Microsoft

No hash list ships in the repository; lists come from your provider at run time. Your calls stay the same when you change the pre-screen.

Testing

moderate runs on SQLite in tests with the fake clock, so clock.advance("24h") ends a hold without waiting. Write the test that matters most first: held content is never readable. See Test your app.

Security and privacy

Held, blocked and removed content is refused by files, media and feeds until cleared.
Hash matches are logged with the list name, never the content.
Every step is audited, and privacy.erase removes a person's reports and statements in the current tenant and region.

FAQ

Is a decision final? No. The author can appeal inside appeals.window (at least 6 months) and a different reviewer resolves it.

Images and video: held outputs.
File uploads: held uploads.
Privacy and data rights: erase a person's reports.

Was this page helpful?