Build a vertical

A "vertical" is an application domain built on top of the platform — AI agents, identity, real estate, or market data. Each vertical reuses the same base capabilities: signed rows, content-addressed primary keys, pipeline manifests, the dataflow runtime, and executor isolation tiers. It contributes its own Rust-native serde schema types, ingestion adapters, transforms, datasets, and optional UI.

The order matters. Schema first, ingestion second, transforms third, service surface fourth, UI fifth, deployment last.

1. Define the Rust-native schema types

Define stored rows, IPC payloads, and long-lived state as serde types in a focused component micro-crate under crates/shared/schema/. Use the component's stable package and public module path, CBOR for row/IPC bytes, and Arrow/Lance for the storage projection. Greenfield identity fields may be required; fields added after release stay optional. The schema remains a checked-in Rust serde contract, with canonical CBOR for stored and transported bytes.

Two things you need from the start: a raw response type (the typed equivalent of whatever JSON or binary payload the source returns) and a per-record type that flattens the response into one row per logical event. If a source returns "an array of N prices at N timestamps in one HTTP response", the raw type holds the whole array exactly as the source returned it, and the per-record type is one price at one timestamp. The DAQ pipeline saves the raw before any transformation runs and the per-record is what the data lake ends up indexing.

Your domain types may also need to participate in the signed primary key system. Reuse the focused primary-key schema crate and the data model at /infrastructure/data; any older generated-schema implementation there is a migration boundary, not a pattern for new application code.

2. Define the dataset reference

Every dataset in the data lake carries a resolvable DatasetReference. Stable, high-frequency platform record classes may use a RecordElementDataType enum variant and its globally unique dataset_name(). Generic or application-owned datasets use the generic record class plus a globally unique type_url, which avoids consuming a central enum slot. The same reference can carry a content hash for content-addressed schema evolution.

The transaction envelope needs enough information to resolve the dataset and schema. Choose the enum path for an established platform class and the type_url path for an ad-hoc or independently evolving application dataset.

3. Ingestion crate (if applicable)

If your vertical pulls data from external APIs, websocket feeds, or file sources, add a collector crate under crates/data/daq/<source>/. Existing collectors to study: weather and real-estate (non-finance domains), macro (REST polling), public-records (file sources), and the market-data collectors under crates/data/daq/ covering REST, websocket-only, and paid authenticated sources.

The implementation contract:

• Define serde structs for the upstream format. Keep a durable raw type when the source payload itself is part of the stored contract.
• Save raw JSONL to disk before any transformation. The DAQ framework provides the writer; use it. This protects you against the source changing its response shape — you can re-parse historical raw files against an updated schema.
• Convert raw to your typed native record, attach a PrimaryKeyId carrying both the source-reported timestamp $t_r$ and the observation timestamp $t_o$, and write through the shared storage layer.
• Implement the DataSource trait so the daq-crawler runtime can drive your collector. See crates/data/daq/weather/src/ingestion.rs for a worked example.

The crawler itself does not featurize data. Its only job is raw retrieval, typed conversion, and write-through.

4. Transforms

Transforms are how your raw rows become useful rows. Choose the tier from the code's trust boundary:

• Built-in operators run in-process for platform-owned operations.
• Trusted Rust batch transforms are content-addressed dynamic libraries loaded through the NATIVE_DYLIB tier.
• Capability-limited WASM UDFs run in process through the default-on Wasmtime sandbox, with fuel, memory, and wall-time bounds and no WASI access.
• Untrusted native and Python functions run through the MICROVM tier on opted-in KVM nodes, using a Firecracker sandbox.

The dataflow executor has no GPU UDF backend. Model serving uses a separate GPU lane. The server request-handler WASM host is also a separate component preview and is not selected by production routes. See the capability matrix and Pipeline Apps runtime placement for the maintained status.

Either way the transform shows up in the dataflow plan as a stage, and the canonical stage ordering is Filter -> Transform -> Aggregate -> Merge -> Sort with backward cycles allowed when iterating. See Data and pipelines for the public execution model.

5. Domain logic crate

Add a crate under the right subgroup. The major divisions are crates/ai/, crates/integrations/, crates/feed/, crates/osint/, and crates/data/identity/ — pick the one that matches your domain or add a sibling if your vertical does not fit existing groupings. Use the crates/template directory as the starting point per CLAUDE.md.

Keep the crate independently buildable. Depend on schema sub-crates (redgold-schema-base, redgold-schema-error) rather than the superset redgold-schema where you can — the dev branch CI builds independent microservices and the superset is for local-dev convenience and release builds. Compile time is a real cost here; the superset graph is enormous.

Set publish = false and rely on .workspace = true for shared deps. New crates almost always want redgold-schema-base.workspace = true at minimum.

6. Declare routes in the pipeline manifest

New HTTP and WebSocket routes are authored under pipelines/routes/<domain>/. Route discovery traverses domain folders recursively. The manifest declares the method, path, authentication requirement, request and response schema types, dataset binding, and executor operation. The stateless edge resolves the route and caller identity, then relays the invocation to the executor. Use DTS HTTP/WS, engine subscriptions, or iroh where ready.

A specialized service crate is appropriate when the application needs a separately operated workload or protocol that the pipeline executor cannot provide. Ordinary application reads, writes, transforms, and subscriptions stay on the manifest path. Managed application artifacts defines the customer-reviewed contract.

Route reachability has two separate gates. The generated V1 route inventory is the authority for self-service routes: today it lists the model API only. Pipeline-app routes are onboarding/managed-preview routes and remain unavailable to a general API-key caller until the repository, dataset, publication, and environment have been approved and provisioned.

7. UI

The Vue 3 UI lives under ui/. Stream has a normal, scrollable chronological conversation lane with the shared composer and status rail. Its optional ambient bubble canvas uses a force-directed, non-scrolling layout inside the canvas. New event views should choose the lane or canvas behavior that matches their content.

8. Deployment

Pipeline applications deploy through the Git-backed build path. During the managed preview, Redgold provisions the dataset and publication boundary, then the normal review and CI path publishes the approved manifest and workload state. Keep the schema, pipeline manifest, transform pin, tests, and application contract in the repository so validation can evaluate the complete change.

9. Docs

Add a section under /applications/<your-vertical>/ covering what the vertical does, which schema types it introduces, which ingestion sources it pulls from, and where its API surface lives. The AI-agents section under docs/content/7.applications/1.ai-agents/ is the model — it covers motivation, the problems the vertical solves, and the implementation walk-through, which is the shape most verticals should follow.

Concrete examples

AI agents are the most active native-schema conversion area, with component crates under crates/shared/schema/ai/ and agent-plane crates under crates/service/agent-controller/, agent-pod-proxy/, and session-babysitter/. Remaining generated-schema paths are legacy migration inventory rather than examples for new work.