---
title: "Import files and pull data"
description: "Read CSV, XLSX, JSON and NDJSON files, map their columns with a confirmed mapping, write rows safely, and pull from outside sources on a schedule."
section: "interfaces"
status: "Building for 0.4 · ships as a beta API"
spec: "imports"
canonical: "https://mainsdk.dev/docs/interfaces/imports"
updated: "2026-10-28"
---

# Import files and pull data

> Read CSV, XLSX, JSON and NDJSON files, map their columns with a confirmed mapping, write rows safely, and pull from outside sources on a schedule.

Status: Building for 0.4 · ships as a beta API · Canonical: https://mainsdk.dev/docs/interfaces/imports · Updated: 2026-10-28

`imports` brings outside data into your app's tables: a person uploads a file, confirms a column mapping, and only then are rows written. The same engine pulls from outside sources on a schedule.

A person uploads a file, the app previews the first rows and suggests a column mapping, the person confirms it, and only then are rows written. The same engine pulls from sources such as a Google Sheet, an Airtable base or a partner's REST API on a schedule. Reach for it when customers hand you spreadsheets or when your data lives in someone else's tool.

It is named `imports` because `import` is a reserved word in JavaScript. File imports and the `rest`, `google-sheets` and `airtable` source drivers are built; Google Sheets and Airtable need a `connections` token, which is not built yet (see below).

## At a glance

- Status: Building for 0.4 · ships as a beta API
- Phase: 1 (Core, public beta)
- Milestone: 0.4 Hardening
- Plan: free in your app (Apache-2.0)
- Spec: https://github.com/hubxero/mainsdk/blob/develop/docs/specs/imports.md
- Category: Data

Calls:

- `imports.preview(file, opts)`
- `imports.file(file, opts)`
- `imports.get(runId)`
- `imports.list(opts?)`
- `imports.cancel(runId, opts)`
- `imports.source(name, opts)`
- `imports.enable(name, opts?)`
- `imports.disable(name)`
- `imports.pull(name, opts?)`
- `imports.conflicts(name, opts?)`

In main.yaml:

```yaml
imports:
  inline-max: 5MB
  batch: 500
  max-rows: 1000000
  max-columns: 500
  json-max: 20MB
  sources:
    crm-sheet: { driver: google-sheets, spreadsheet: "1AbC…", range: "Contacts!A:F", auth: connection:google, every: 15m }
    partner-api:
      driver: rest
      url: https://api.partner.com/v1/items
      auth: secret:PARTNER_TOKEN
      every: 5m
      full-every: 24h
      deletes-guard: 50%
```

Guarantees, whatever the driver:

- Rows are written only after the person confirms the mapping
- A repeated run with the same file, target, key and mapping writes nothing twice
- Deletions from a source happen only on a full pass, behind a guard
- An unsafe workbook is refused before its rows are read


## Drivers

Pick one in main.yaml with `imports.driver`.

| Driver | Id in main.yaml | Default | Note |
| --- | --- | --- | --- |
| Built in parsers for CSV, XLSX, JSON and NDJSON | `builtin` | yes |  |
| Google Sheets | `google-sheets` |  |  |
| Airtable | `airtable` |  |  |
| REST API with a cursor | `rest` |  |  |

## Codes

| Code | Where | Meaning |
| --- | --- | --- |
| `MAIN-IMPORT-0020` | check | A source under `imports.sources` has no `imports.source()` in the code, or the code defines a source `main.yaml` doesn't declare |
| `MAIN-IMPORT-0001` | runtime | The format or encoding isn't supported: not CSV, XLSX, JSON or NDJSON, not UTF-8, a JSON value that isn't an array, a legacy `.xls` or an encrypted workbook (415) |
| `MAIN-IMPORT-0002` | runtime | The file is over a limit: `max-rows`, `max-columns`, `json-max` or the `files` size (413); rows up to the limit are not written |
| `MAIN-IMPORT-0003` | runtime | The mapping is incomplete or unconfirmed: a required field has no column, a mapped column doesn't exist, or a field would come from a model suggestion not passed back in `mapping` (422) |
| `MAIN-IMPORT-0004` | runtime | The file isn't `ready` yet (still scanning) (409, retryable) |
| `MAIN-IMPORT-0005` | runtime | The run stopped because failed rows passed `maxErrors`; the report lists them (422) |
| `MAIN-IMPORT-0006` | runtime | An unsafe workbook: a zip entry over the size or ratio limit, a path outside the archive, or XML with a `DOCTYPE` or entity (422) |
| `MAIN-IMPORT-0010` | runtime | A pull failed: the source was unreachable or refused; the cursor is unchanged and the next run retries |
| `MAIN-IMPORT-0011` | runtime | The deletion guard tripped: a full pass returned fewer keys than `deletes-guard`, so nothing was deleted |
| `MAIN-IMPORT-0012` | runtime | The source rejected the stored cursor; a full pass is scheduled |

## Features

| Feature | Phase | Milestone | Spec ticket |
| --- | --- | --- | --- |
| Read CSV, XLSX, JSON and NDJSON | 1 | 0.4 | `imports#read` |
| Map columns with a confirmed mapping | 1 | 0.4 | `imports#mapping` |
| Dry run with a row error report | 1 | 0.4 | `imports#dry-run-report` |
| Resumable batched writes | 1 | 0.4 | `imports#upsert-resume` |
| Large files run as a workflow with progress | 1 | 0.4 | `imports#workflow-progress` |
| Excel workbooks with safety limits | 1 | 0.4 | `imports#xlsx` |
| Scheduled pulls from sources | 1 | 0.4 | `imports#pull` |
| Conflict rules and guarded deletes | 1 | 0.4 | `imports#conflicts-deletes` |
| Google Sheets and Airtable sources | 1 | 0.4 | `imports#drivers-sheets-airtable` |
| Checks for undeclared sources | 1 | 0.4 | `imports#checks` |

## MainSDK Cloud

Everything on this page is free in your app.


## Quick start

```ts title="app/admin/import/actions.ts"
"use server";
import { imports } from "mainsdk";
import type { FileRef } from "mainsdk/files";
import { s } from "mainsdk/schema";

const Contact = s.object({ email: s.string(), name: s.string() });

export async function previewContacts(file: FileRef) {
  return imports.preview(file, { schema: Contact, key: "email" });
}

export async function importContacts(file: FileRef, mapping: Record<string, "email" | "name" | null>) {
  return imports.file(file, { schema: Contact, key: "email", into: "contacts", mapping });
}
```

`FileRef` is the upload from [`files`](https://mainsdk.dev/docs/interfaces/files). The `mapping` you pass to `imports.file` is the confirmation: nothing is written until it is there.

## Configuration

```yaml title="main.yaml"
imports:
  inline-max: 5MB               # larger files, or more than 10 000 rows, run as a workflow
  batch: 500                    # rows per upsert batch and per workflow step
  max-rows: 1000000
  max-columns: 500
  json-max: 20MB                # a JSON array is read whole; NDJSON streams at any size
  suggest: { samples: 3 }       # values per column sent to the model after redaction
  sources:
    crm-sheet:   { driver: google-sheets, spreadsheet: "1AbC…", range: "Contacts!A:F", auth: connection:google, every: 15m }
    partner-api:
      driver: rest
      url: https://api.partner.com/v1/items
      auth: secret:PARTNER_TOKEN
      cursor: { param: since, next: $.next }
      items: $.items
      every: 5m
      full-every: 24h           # the only time deletions are detected
      deletes-guard: 50%        # a full pass returning fewer keys than this deletes nothing
```

| Key | What it does |
| --- | --- |
| `inline-max`, `batch` | When a file runs as a workflow, and how many rows each write takes |
| `max-rows`, `max-columns`, `json-max` | Hard limits. A file over one is refused with `MAIN-IMPORT-0002` and no rows are written |
| `suggest.samples` | How many values per column go to the model. `0` sends headers only |
| `sources` | Outside sources to pull from, each with a `driver`, `auth` and `every` |

Source credentials go in as `secret:` or `connection:` references. A source's `auth: connection:google` uses the person's connection from [`connections`](https://mainsdk.dev/docs/interfaces/connections).

## What you get

One `imports` object with two jobs: file imports (`preview`, `file`) and scheduled pulls from named sources (`source`, `enable`, `pull`). Runs, counts and failed-row reports are kept per tenant.

### Calls

`import { imports } from "mainsdk"` (or `mainsdk/imports`) is built on first use from the `imports` section.

| Call | What it does |
| --- | --- |
| `imports.preview(file, opts)` | Read the first rows and return the columns, a suggested mapping and the required fields no column maps to |
| `imports.file(file, opts)` | Import a file with a confirmed mapping. Small files finish inline; larger ones return `state: "running"` and continue in a workflow |
| `imports.get(runId)` | Read one run: its state and counts |
| `imports.list(opts?)` | List the current tenant's runs, newest first, with `cursor` and `limit` |
| `imports.cancel(runId, { reason })` | Stop a run after the batch in flight. Rows already written are kept |
| `imports.source(name, opts)` | Define a pull source in code; it must match the source in `main.yaml` |
| `imports.enable(name, opts?)` | Start pulling a source on its schedule |
| `imports.disable(name)` | Stop pulling a source. Its earlier rows stay |
| `imports.pull(name, opts?)` | Pull a source now, outside the schedule |
| `imports.conflicts(name, opts?)` | List rows where the source and the app both changed |

## Examples

### Preview, then import

```ts title="scripts/import-once.ts"
import { imports } from "mainsdk";
import type { FileRef } from "mainsdk/files";
import { s } from "mainsdk/schema";

const Contact = s.object({ email: s.string(), name: s.string() });

declare const file: FileRef;   // the upload

const preview = await imports.preview(file, { schema: Contact, key: "email" });
console.log(preview.columns, preview.suggested, preview.missing);

const run = await imports.file(file, {
  schema: Contact, key: "email", into: "contacts",
  mapping: { "E-mail": "email", "Full name": "name" },
  dryRun: true,                 // report what would change; write nothing
});
console.log(run.counts, run.report);
```

`preview.missing` lists required fields no column maps to. `run.report` is a CSV of failed rows, stored with [`files`](https://mainsdk.dev/docs/interfaces/files).

### Pull from a source on a schedule

```ts title="jobs/sources.ts"
import { imports } from "mainsdk";
import { s } from "mainsdk/schema";

const Contact = s.object({ email: s.string(), name: s.string() });

export const crm = imports.source("crm-sheet", {
  schema: Contact, key: "email", into: "contacts",
  mapping: { "E-mail": "email", "Full name": "name" },
  conflict: "source-wins",      // source-wins | app-wins | newest | flag
  deletes: "soft",              // ignore | soft | hard
});

await imports.enable("crm-sheet");
await imports.pull("crm-sheet");   // run now, outside the schedule
```

Each source is declared once in `main.yaml` and defined once in code. `mainsdk check` fails when the two do not match (`MAIN-IMPORT-0020`).

### Read the conflicts

```ts
import { imports } from "mainsdk";

const page = await imports.conflicts("crm-sheet", { limit: 50 });
```

A conflict is a row where the source and the app both changed. `flag` leaves it for a person to decide.

## What is connected, and what is not

`app.imports` is wired to `files`, tenancy, `workflows`, `jobs`, the log, `http` behind `network`, secrets, `ai` for `suggest: "ai"` (using the first non-pattern model of `ai.models`) and the app's `db` for model targets, which is only opened when a run writes a model. Run records live in `db` collections when the app has a `db` section, otherwise in process. These are not built yet, so set `app.socialPorts.imports = { ... }` before the first use if you have your own:

| Port | What it does | Without it |
| --- | --- | --- |
| `realtime` | Publishes progress events on a run's channel | Read progress with `imports.get(runId)` |
| `connections` | Gives a source `auth: connection:<provider>` the person's token | Google Sheets and Airtable sources end as a failed run; `secret:` auth still works |

## Errors and checks

| Code | When | Fix |
| --- | --- | --- |
| `MAIN-IMPORT-0001` | Format or encoding not supported (not UTF-8, legacy `.xls`, encrypted workbook) | Convert the file to CSV, XLSX, JSON or NDJSON in UTF-8 |
| `MAIN-IMPORT-0002` | The file is over a limit | Split the file, or raise the limit under `imports` |
| `MAIN-IMPORT-0003` | A required field has no column, or the mapping is unconfirmed | Map every required field and confirm the mapping |
| `MAIN-IMPORT-0005` | Failed rows passed `maxErrors`; the run stopped | Fix the rows in the report, then run again |
| `MAIN-IMPORT-0006` | An unsafe workbook (zip bomb, path escape, XML entity) | Save the workbook again |
| `MAIN-IMPORT-0020` | A source in `main.yaml` has no code, or the other way round | Declare it in both places |

`MAIN-IMPORT-0004`, `0010`, `0011` and `0012` are runtime codes; see [check codes](https://mainsdk.dev/docs/reference/check-codes).

## Swapping drivers

Files are read by the built-in parsers, so there is no driver to swap for them. Each source names its driver in `main.yaml`: `google-sheets`, `airtable` or `rest`. Changing a source's driver changes only that source's block; your `imports.source()` code stays the same.

## Testing

`createTestApp` builds `imports` over the same fakes as the rest of the app: an in-memory `files` store, the fake clock and the local job queue. Preview and import a small CSV in a test, and assert on `run.counts` and `run.report`. See [Test your app](https://mainsdk.dev/docs/guides/testing).

## Security and privacy

- Rows are written only after the mapping is confirmed.
- Values sent to a model for column suggestions are redacted first, and `suggest.samples: 0` sends headers only.
- A full pass that returns too few keys deletes nothing (`deletes-guard`), so a broken source cannot empty your table.
- Workbooks are checked for zip bombs and XML entities before any row is read.

## Next

- [Files](https://mainsdk.dev/docs/interfaces/files): where uploads and reports live.
- [Connections](https://mainsdk.dev/docs/interfaces/connections): give a source the person's own account.
- [Workflows](https://mainsdk.dev/docs/interfaces/workflows): the engine behind large imports.
- [Import a spreadsheet guide](https://mainsdk.dev/docs/guides/import-data): a first import, end to end.