Import your data

rac des import imports JSONL or Parquet into an existing dataset with a declared schema. It validates each row against that schema, encodes it as native CBOR and submits sequential transactions through the normal data-engine write path. The CLI does not create a dataset or infer and publish a schema from a file.

Prepare the dataset and connection

Use the dataset name, schema, engine endpoint and caller credentials provisioned for your project. The importer needs DATA_ENGINE_URL, an authorized engine credential and the matching caller identity. An account key does not grant access to another owner's dataset. Keep connection credentials in the project's secret manager.

Inspect the declaration through your configured engine connection:

rac dataset describe my_dataset

Field names and values must match the registered schema. The importer rejects unknown fields, missing required values and incompatible types. For a new record shape, define its checked-in Rust serde schema and dataset first. During the managed preview, initial provisioning remains coordinated. See Build a vertical or Managed application artifacts.

Import a file or directory

rac des import records.jsonl --dataset my_dataset \
  --batch-size 500 --cursor records.import-cursor.json

Replace the input, dataset and cursor with project-specific values. The command also accepts a .parquet file or a directory of .jsonl and .parquet files. Directory input is non-recursive, sorted by filename and limited to 256 files. Keep the cursor outside version control and use a separate cursor for each intentional import.

Batch size is 1–1000 rows, with 500 as the default. The importer also bounds encoded rows to 4 MiB and pending row payloads to 8 MiB per batch. It submits batches sequentially and checkpoints acknowledged progress. Progress and the final JSON report include scanned rows, acknowledged rows and batches, error counts and up to 20 error examples.

By default, invalid rows are collected and skipped while valid rows continue; a run with row errors still returns an error. Add --strict to stop at the first invalid row. Strict mode does not roll back earlier acknowledged batches. Inspect both the exit status and the error report before calling an import complete.

Resume and uncertain writes

Rerun the same command with the same input and cursor. The cursor binds the input files and contents, schema, endpoint, caller, batch size and strict-mode choice. A mismatch is rejected. Resume skips rows covered by saved progress; it does not retry the rows recorded as validation errors. Corrected input needs an intentional recovery plan and a new cursor, accounting for rows already written.

A crash after the engine accepts a batch but before the cursor is saved can cause duplicate rows on replay. The importer does not guarantee exactly-once writes. Reconcile an uncertain submission against the destination before retrying; deleting the cursor can replay the entire import. Keep the original cursor and input when investigating a lost response.

Verify the result

Read through the same caller's authorized dataset connection:

rac dataset head my_dataset --rows 5 --json
rac des sql "select * from my_dataset limit 5"

Both commands perform bounded reads. Compare the schema, expected row values and the importer's acknowledgement/error totals. An acknowledgement establishes an accepted transaction; a sample does not prove that every source row arrived. For application-specific imports, also check the application's declared read route.

rac dataset diff compares bounded owner-scoped row sets, including historical views when supported:

rac dataset diff --left my_dataset --right expected_dataset --limit 100 --json

Use --key for a unique logical record key and --as-of-left or --as-of-right for supported historical selectors. The command compares key sets in the bounded reads and checks values for at most 1000 shared keys, selected with a repeatable seed. Its report identifies the sample and read window. It does not repair data or establish equality beyond the sampled values and retrieved keys.

Format support

Other write paths

For recurring application writes, use a managed app's declared upsert route. Its schema, transform and route stay reviewable in Git. For programmatic imports, construct signed native transactions through the SDK and submit them to POST /api/query/transaction/submit; the engine checks the signature over canonical CBOR. A collector is appropriate for ongoing external API or feed backfills; see Build a vertical.

rac des submit remains limited to its declared internal benchmark datasets. Use rac des import, an application action or the signed transaction API for the applicable dataset. MCP clients can discover supported installed writes through the data tools; the file importer itself is a local CLI workflow.