Data and pipelines

The platform data layer is the runtime foundation for pipeline applications. It stores typed application records, executes declared transforms, and keeps agent activity available as a supporting dataset for authoring, history, and operational review.

Datasets

A dataset is a named collection of typed records. Records retain source and ingestion metadata so callers can distinguish when an event occurred from when the platform received it. Hot query paths use Postgres-backed services, while larger analytical collections can use columnar storage.

Pipelines

Pipelines describe how records enter the platform, how transforms run, and how a result is returned or stored. The platform uses these contracts for product features as well as application-specific work.

Agent history

Conversations, messages, tool activity, usage, and lifecycle events can be stored as structured records alongside application data. This makes a session inspectable and queryable while agents help author and operate pipeline applications. It does not make model generation deterministic; a reproducible evaluation also needs pinned code, data, model, and configuration.

Public availability

The data engine and pipeline system are active parts of the hosted platform. Their broad generic execution and authoring interfaces are still under development as external contracts, so they are excluded from the public API inventory. Supported endpoints will appear in the generated public API reference only after they have reviewed schemas, examples, and compatibility expectations.