Pipeline manifest preview

A Redgold pipeline manifest is TOML that describes a named graph of inputs, transforms, outputs, and optional HTTP entry points. CI parses the manifest, validates it, and lowers it to the signed runtime specification used by the executor.

This Developer Preview covers the authoring shape used by onboarded application repositories. Publication and deployment remain managed and approval-gated.

The three layers

Application authors normally edit the manifest. CI owns generated signatures, hashes, timestamps, and deployment state.

The current pipeline content hash and author signature cover the canonical CBOR encoding of nodes, output_node_ids, and spec_name. version_tag, description, tags, latest, and timestamps are registry/display metadata outside that identity. Resolve an exact revision with spec_name and spec_hash. The portable AppRelease contract adds a separate whole-package hash and signature boundary over schemas, pipelines, surfaces, assets, capabilities, and dependencies. Local builds currently emit unsigned release candidates; registry signature verification is planned.

Smallest useful manifest

name = "ticket-summary"
description = "Create a compact summary when a ticket changes."
tags = ["tickets", "preview"]
family = "tickets"
role = "materialization"
origins = ["ticket-events"]
maturity = "preview"

[[flow]]
transform = "ticket-normalize"
in = "ticket"
out = "ticket_normalized"

[[flow]]
transform = "ticket-summarize"
out = "ticket_summary"

Each [[flow]] step consumes the previous step's output. Use independent [[nodes]] entries for fan-out or multiple input leaves.

Reviewed authoring subset

Manifest fields

All four typed metadata fields are optional. Existing family:*, role:*, and origin:* tags remain accepted. Typed fields take precedence, and catalog inference still supplies family and role when neither form is present.

Choose one of flow, nodes, or route in one manifest. Node identifiers must be unique after lowering.

Transform node fields

Transform bindings can run continuously, on a schedule, or manually. Managed onboarding chooses the mode and required capabilities when the short form is insufficient.

Inline typed dataflow

For common row operations, a node can carry the complete dataflow directly in TOML. This is the read path from the collaborative-board fixture:

[[nodes]]
node = "read"

[nodes.dataflow]
limit = 500

[nodes.dataflow.source]
kind = "dataset"
dataset = "app_example.collaborative_board.Card"

[[nodes.dataflow.stages]]
kind = "filter"

[[nodes.dataflow.stages.must]]
column = "archived"
op = "eq"
value = false

[[nodes.dataflow.stages]]
kind = "sort"
columns = ["column_id", "title"]
direction = "ascending"

[[nodes.dataflow.stages]]
kind = "project"
op = "select"
columns = ["id", "column_id", "title", "labels"]

[nodes.dataflow.sink]
kind = "http_response"

The current strict authoring catalog covers filter, project, aggregate, sort, window, distinct, partitioned_limit, http_fetch, and an execution-boundary exchange. A transform stage applies declared builtin functions, operator names a typed engine factory, and function pins a custom artifact. Each stage lowers into the checked-in Rust Dataflow model used by the executor; misspelled fields and unavailable stage kinds fail during parsing. Analytic windows can participate in the ordinary batch graph. Current durable streaming authoring requires exactly window followed by aggregate, supports bounded tumbling/sliding shapes, and rejects session windows, watermark hints, nonzero lateness, excessive sliding fan-out, and unsupported aggregations.

Compact HTTP routes

The top-level [route] form describes the common single-node HTTP dataflow. HTTP fields live directly on route, an inline source is supplied by default, and [[route.stages]] preserves executor order:

name = "ticket-list"
description = "List tickets owned by the authenticated workspace."
family = "tickets"
role = "route"

[route]
method = "POST"
path = "/v1/tickets/list"
protocol = "unary"
requires_auth = true
route_scope = "public"
match_kind = "exact"
capabilities = ["data_engine"]

[[route.stages]]
operator = "internal.data_engine_execute"

[route.stages.internal_rpc]
mode = "data_engine_execute"
payload_column = "request"

[route.stages.internal_rpc.dataset_read]
dataset = "tickets_v1"

[[route.stages.internal_rpc.dataset_read.stages]]
kind = "filter"
must = [{ column = "completed", value = false }]

[[route.stages]]
operator = "response.unwrap_de_rows"

operator = "factory.name" is stage sugar for kind = "operator" plus name = "factory.name"; the expanded form remains accepted. Routes that need something beyond the inline-source default can declare [route.source] and [route.sink]. Dataflow-level limit, offset, stream_mode, [[route.param_overlay]], and [route.resource_limit_overrides] also live under route.

route_scope = "public" controls edge reachability. It does not publish an API contract. Public reference inclusion requires a separate reviewed public-api tag, complete documentation metadata, and generator checks.

Validation lifecycle

Repository automation performs these checks before a manifest can progress:

1. Parse TOML with unknown-field rejection.
2. Lower the authoring form into the runtime graph.
3. Verify unique nodes, valid edges, and resolvable functions.
4. Check route scope, authentication, and capability declarations.
5. Compare generated artifacts with the committed versions.
6. Run repository tests and request approval for publication or deployment.

A parse or lowering failure should be corrected in the manifest. Generated runtime files should be regenerated through the repository workflow.

Public route, version, and service expectations

route_scope = "public" describes intended edge reachability for an onboarded application. It does not make a route self-service. The generated V1 route inventory is the public contract for general API-key callers; pipeline-app routes become callable only after managed onboarding, dataset provisioning, review, publication, and deployment.

Pin application behavior to an exact pipeline revision (spec_name plus spec_hash) when reproducibility matters. Treat latest and human version_tag values as mutable registry/display metadata. Publish a new revision for behavior or schema changes, record migration guidance for stored data, and keep old revisions available for the installation's rollback policy. Deprecation and removal are managed release decisions; a route should remain in the generated public reference while it is supported.

The public V1 inventory currently contains the model API routes and does not provide a generic pipeline-app health endpoint. During managed deployment, use the onboarding deployment status surface and a bounded, owner-scoped smoke request. Treat authentication, validation, and contract-shape failures as application or release signals; retry only statuses documented by the API contract.

Current limits

• Manifest publication is available to onboarded repositories.
• The public preview does not expose signing fields or deployment topology.
• Inline relational dataflow bodies use a strict, implemented projection into the checked-in Rust Dataflow. Unknown keys and incompatible stage shapes fail during parsing.
• The strict TOML compiler currently rejects two-input merge, join, and set_op stages because the normal executor request carries no independent right-hand input branch. It also rejects stage-level limit until the standard executor has a compatible backend. Top-level output limit remains available.
• exchange currently marks an execution boundary. Physical partitioning strategy fields are rejected until the runtime applies them.
• Unsupported operators fail validation instead of being ignored.
• Direct route consumers should rely on the generated public API reference, not the manifest alone.

Continue with the data platform preview or the tickets application walkthrough.