Import files and pull data

imports brings outside data into your app's tables: a person uploads a file, confirms a column mapping, and only then are rows written. The same engine pulls from outside sources on a schedule.

A person uploads a file, the app previews the first rows and suggests a column mapping, the person confirms it, and only then are rows written. The same engine pulls from sources such as a Google Sheet, an Airtable base or a partner's REST API on a schedule. Reach for it when customers hand you spreadsheets or when your data lives in someone else's tool.

It is named imports because import is a reserved word in JavaScript. File imports and the rest, google-sheets and airtable source drivers are built; Google Sheets and Airtable need a connections token, which is not built yet (see below).

imports:
  inline-max: 5MB
  batch: 500
  max-rows: 1000000
  max-columns: 500
  json-max: 20MB
  sources:
    crm-sheet: { driver: google-sheets, spreadsheet: "1AbC…", range: "Contacts!A:F", auth: connection:google, every: 15m }
    partner-api:
      driver: rest
      url: https://api.partner.com/v1/items
      auth: secret:PARTNER_TOKEN
      every: 5m
      full-every: 24h
      deletes-guard: 50%

import { imports } from "mainsdk/imports";
import { Contact } from "./schema";

const preview = await imports.preview(file, { schema: Contact, key: "email" });
const run = await imports.file(file, {
  schema: Contact, key: "email", into: "contacts",
  mapping: { "E-mail": "email", "Full name": "name" },
});

imports:
  driver: builtin

Quick start

"use server";
import { imports } from "mainsdk";
import type { FileRef } from "mainsdk/files";
import { s } from "mainsdk/schema";

const Contact = s.object({ email: s.string(), name: s.string() });

export async function previewContacts(file: FileRef) {
  return imports.preview(file, { schema: Contact, key: "email" });
}

export async function importContacts(file: FileRef, mapping: Record<string, "email" | "name" | null>) {
  return imports.file(file, { schema: Contact, key: "email", into: "contacts", mapping });
}

FileRef is the upload from files. The mapping you pass to imports.file is the confirmation: nothing is written until it is there.

Configuration

imports:
  inline-max: 5MB               # larger files, or more than 10 000 rows, run as a workflow
  batch: 500                    # rows per upsert batch and per workflow step
  max-rows: 1000000
  max-columns: 500
  json-max: 20MB                # a JSON array is read whole; NDJSON streams at any size
  suggest: { samples: 3 }       # values per column sent to the model after redaction
  sources:
    crm-sheet:   { driver: google-sheets, spreadsheet: "1AbC…", range: "Contacts!A:F", auth: connection:google, every: 15m }
    partner-api:
      driver: rest
      url: https://api.partner.com/v1/items
      auth: secret:PARTNER_TOKEN
      cursor: { param: since, next: $.next }
      items: $.items
      every: 5m
      full-every: 24h           # the only time deletions are detected
      deletes-guard: 50%        # a full pass returning fewer keys than this deletes nothing

Source credentials go in as secret: or connection: references. A source's auth: connection:google uses the person's connection from connections.

What you get

One imports object with two jobs: file imports (preview, file) and scheduled pulls from named sources (source, enable, pull). Runs, counts and failed-row reports are kept per tenant.

Calls

import { imports } from "mainsdk" (or mainsdk/imports) is built on first use from the imports section.

Examples

Preview, then import

import { imports } from "mainsdk";
import type { FileRef } from "mainsdk/files";
import { s } from "mainsdk/schema";

const Contact = s.object({ email: s.string(), name: s.string() });

declare const file: FileRef;   // the upload

const preview = await imports.preview(file, { schema: Contact, key: "email" });
console.log(preview.columns, preview.suggested, preview.missing);

const run = await imports.file(file, {
  schema: Contact, key: "email", into: "contacts",
  mapping: { "E-mail": "email", "Full name": "name" },
  dryRun: true,                 // report what would change; write nothing
});
console.log(run.counts, run.report);

preview.missing lists required fields no column maps to. run.report is a CSV of failed rows, stored with files.

Pull from a source on a schedule

import { imports } from "mainsdk";
import { s } from "mainsdk/schema";

const Contact = s.object({ email: s.string(), name: s.string() });

export const crm = imports.source("crm-sheet", {
  schema: Contact, key: "email", into: "contacts",
  mapping: { "E-mail": "email", "Full name": "name" },
  conflict: "source-wins",      // source-wins | app-wins | newest | flag
  deletes: "soft",              // ignore | soft | hard
});

await imports.enable("crm-sheet");
await imports.pull("crm-sheet");   // run now, outside the schedule

Each source is declared once in main.yaml and defined once in code. mainsdk check fails when the two do not match (MAIN-IMPORT-0020).

Read the conflicts

import { imports } from "mainsdk";

const page = await imports.conflicts("crm-sheet", { limit: 50 });

A conflict is a row where the source and the app both changed. flag leaves it for a person to decide.

What is connected, and what is not

app.imports is wired to files, tenancy, workflows, jobs, the log, http behind network, secrets, ai for suggest: "ai" (using the first non-pattern model of ai.models) and the app's db for model targets, which is only opened when a run writes a model. Run records live in db collections when the app has a db section, otherwise in process. These are not built yet, so set app.socialPorts.imports = { ... } before the first use if you have your own:

Errors and checks

MAIN-IMPORT-0004, 0010, 0011 and 0012 are runtime codes; see check codes.

Swapping drivers

Files are read by the built-in parsers, so there is no driver to swap for them. Each source names its driver in main.yaml: google-sheets, airtable or rest. Changing a source's driver changes only that source's block; your imports.source() code stays the same.

Testing

createTestApp builds imports over the same fakes as the rest of the app: an in-memory files store, the fake clock and the local job queue. Preview and import a small CSV in a test, and assert on run.counts and run.report. See Test your app.

Security and privacy

• Rows are written only after the mapping is confirmed.
• Values sent to a model for column suggestions are redacted first, and suggest.samples: 0 sends headers only.
• A full pass that returns too few keys deletes nothing (deletes-guard), so a broken source cannot empty your table.
• Workbooks are checked for zip bombs and XML entities before any row is read.

Next

• Files: where uploads and reports live.
• Connections: give a source the person's own account.
• Workflows: the engine behind large imports.
• Import a spreadsheet guide: a first import, end to end.