vs Apache Spark
Apache Spark is a distributed analytics engine. It executes batch, streaming, SQL, machine-learning, and graph workloads through APIs in Python, SQL, Scala, Java, and R. Applications bring their own storage, serving layer, deployment system, and operational tooling around the compute engine.
Redgold includes a dataflow engine as one part of an application platform. A typed schema and pipeline manifest connect ingestion, transforms, materialized datasets, HTTP or WebSocket routes, a GitOps deployment, and the coding agents that operate them. Its Python surface includes a PySpark-shaped compatibility layer, while the execution and storage path remains Redgold's own.
Criteria
| Criterion | Redgold | Apache Spark |
|---|---|---|
| Product unit | A pipeline application spanning schema, compute, storage, routes, agents, and deployment | A distributed analytics application or query job |
| Authoring | Rust serde schemas encoded with CBOR, pipeline manifests, Rust transforms, and a PySpark-shaped client surface | Python, SQL, Scala, Java, or R APIs across the Spark ecosystem |
| Execution | Dataflow stages resolved by the executor, plus compute-on-write transforms | Distributed execution graphs for batch and structured streaming |
| Storage | Lance/Parquet cold storage and a Postgres hot tier under one addressing model (data and pipelines) | Reads and writes external data systems; storage is outside the engine |
| Serving | Declared HTTP and WebSocket routes dispatch into application pipelines | Application serving is assembled separately from Spark jobs |
| Deployment | Git-backed container builds, Kubernetes, and ArgoCD (managed deployment) | Cluster deployment depends on the Spark environment and surrounding platform |
| AI and agents | Managed coding-agent workspaces and model API routing share the application runtime | Outside Spark's scope |
| Current scale | Early single-cluster production use by one operator | Established distributed analytics ecosystem |
Redgold scope
Spark corresponds to the data-processing layer of the Redgold stack. Redgold's product boundary includes the records before a transform, the datasets and routes after it, the agent that edits the application, and the deployment that runs it. The fixed artifact set gives CI and automated review a bounded application shape to validate.