Import your data

Every other onboarding path assumes data is born inside the platform: a collector pulls it, an agent produces it, a pipeline writes it. This page is for the other case — you already have a dataset (a directory of JSONL, a Parquet export, a Postgres table) and want it queryable here.

The honest state today: there is no first-class bulk file-upload route. You do not POST a CSV and get a dataset back, and there is no generic rac des import <file> verb. What exists is the write path every in-platform producer already uses — typed rows through the data engine — plus the query surface to read them back. Importing your data means mapping it onto a checked-in Rust serde record and driving it through one of the write paths below. That mapping is not overhead you can skip: the engine stores typed, content-addressed rows, not opaque blobs, so a schema is the price of entry and also what makes the data queryable, prunable, and joinable afterward.

The biggest gap — a first-class file-import route so a tenant can hand the platform a Parquet or JSONL file directly — is tracked as planned work; this page documents what runs now.

The shape of an import

Whichever write path you pick, the middle step is the same and is the real work: turn each source record into one Rust serde record.

Follow the raw-plus-per-record split described in Build a vertical. One raw serde type mirrors the source payload as-is (a whole CSV row, one JSON object, one Parquet row group's logical record); a second, per-record message flattens it into exactly one row per logical event. If your source hands back "an array of readings at N timestamps in one object", the raw message keeps the array and the per-record message is one reading at one timestamp. Add these under crates/shared/schema/<yourdomain>/. Everything downstream — the write, the partition pruning, the query — keys off that per-record type.

Path A — define a managed app with an upsert route (recommended)

For a dataset you own and will keep writing to, define the record shape, transform, and narrow read/write routes as one managed application contract. Managed application artifacts describes the schema, pipeline TOML, transform, tests, and API contract that stay visible in Git. During the preview, Redgold coordinates the initial repository shape and dataset name with the customer.

This is the right choice when the import is recurring or the dataset is yours long-term: you get a stable route, a named dataset, and the read surface for free.

Path B — the general signed-transaction write

The platform-wide write chokepoint is POST /api/query/transaction/submit. It takes a canonical-CBOR-encoded, signed SubmitTransactionRequest — your rows ride inside a Transaction as TransactionRecordElements, one per record, each carrying a PrimaryKeyId. This is the path the UI and every in-platform producer use, and it is available now without a deploy step, but it is a programmatic path: you construct and sign the transaction with the SDK or native client because the engine verifies the signature over the exact CBOR bytes.

Reach for this when you are writing a one-off batch from your own code and do not want to stand up an app. The record-element registration and PrimaryKeyId construction are the same ones documented in Build a vertical.

Path C — a collector, for recurring external sources

If the data is not a static file you hold but a live external source (an API, a feed, a websocket) you will pull repeatedly, write a collector under crates/data/daq/<source>/ instead of importing by hand. That is the Rust schema → raw JSONL → typed record → Lance path in Build a vertical §3, and it gives you backfill plus live ingestion in one dataset. This is import as an ongoing pipeline rather than a one-time load.

Verify the write landed

In a managed Redgold development environment, the operator can authenticate as the maintained public synthetic principal, list its datasets, and query rows back:

eval "$(rac auth jwt --role public)"   # mints DATA_ENGINE_JWT for the redgold.ai audience
rac des datasets                        # datasets your principal owns
rac des sql "select * from <your_dataset> limit 1"

rac des datasets is owner-scoped: a fresh principal that has not written anything sees {"datasets": []}, which is the expected empty state, not an error. rac des sql lowers a SELECT to a dataflow and runs it against the FROM dataset over the DES request adapter, rendering rows as JSON — this is the reliable read-back and the one to use in a verification loop. Projection-style rac des query --dataset ... exists as well, but resolves a narrower set of datasets, so prefer rac des sql when confirming an import.

rac auth jwt --role public uses repository-managed browser state for the synthetic public test identity. It is an operator verification command rather than customer API-key authentication. Customers verify an onboarded dataset through the approved application's read route after managed publication.

Format matrix

None of the four common source formats has a first-class one-command import today. Each maps onto the paths above; the difference is how much glue the format costs before it becomes typed rows.

The through-line: every cell routes to "map to a per-record Rust type, then write via Path A or B." The schema authoring is the actual import work; the write mechanics are already there.

Planned: first-class file import

The missing piece is a route that takes a file — Parquet or JSONL first — plus a target dataset and does the map-and-write for you, so a tenant with an existing export does not have to scaffold an app or write an SDK program. This is filed as a backlog issue (2026-07-17-first-class-file-import-route); until it lands, the paths above are the supported ways in.