vs Apache DataFusion
Apache DataFusion is an embeddable Rust query engine built on Arrow. It provides SQL and DataFrame APIs, physical and logical planning, common file-format support, and extension points for building a data-centric system. The host application supplies storage policy, network APIs, deployment, tenancy, and operations.
Redgold is the host platform rather than a query-engine library. Its Rust serde and CBOR dataflow contract runs through a dedicated executor and data engine, with Lance/Parquet and Postgres storage, declared application routes, sandboxed untrusted code, and GitOps deployment. Redgold does not expose DataFusion's session or extension APIs as its application contract.
Criteria
| Criterion | Redgold | Apache DataFusion |
|---|---|---|
| Product unit | An operated pipeline application | A Rust library embedded in another system |
| Query interface | CBOR-backed Rust serde dataflow plans, data APIs, SQL surfaces, and a PySpark-shaped client | SQL and DataFrame APIs over Arrow |
| Planner and runtime | Redgold's dataflow planner, executor tiers, and engine | Extensible logical and physical query planning inside the host process |
| Storage | Managed Lance/Parquet cold tier and Postgres hot tier (data and pipelines) | Connectors and file-format support; lifecycle and storage policy belong to the host |
| Application serving | Declared edge routes resolved to pipelines and executor-run code | Supplied by the host application |
| Untrusted code | Capability-limited Wasmtime for WASM UDFs; deny-by-default Firecracker path for native/Python code on opted-in nodes | Isolation policy belongs to the embedding application |
| Deployment and operations | Container build, Kubernetes, ArgoCD, observability, and agent operations | Supplied by the embedding application |
| AI and agents | Coding-agent fleet and model routing integrated with the same repositories and data | Outside DataFusion's scope |
Redgold scope
DataFusion corresponds to an engine component that a larger system can embed. Redgold defines and operates that larger system: schemas, ingestion, storage tiers, transformations, routes, isolation, agent workspaces, and deployments all share one application contract.