Control Plane vs. Compute Plane in Data Pipelines

Define the Two Planes

The terms come from networking, where the control plane decides how traffic should be routed and the data plane carries out that routing. Software architecture borrowed the same split and applied it well beyond routers.

Kubernetes is the version most engineers already know. Its control plane runs the API server, the scheduler, and the controllers that decide what should happen across a cluster, while worker nodes carry out those decisions by running the actual containers.^[1] Databricks draws the line the same way: its control plane hosts the web application, job scheduling, and workspace metadata inside Databricks' own account, while the compute plane, the clusters that read, transform, and write data, typically runs inside the customer's own cloud account instead.^[2]

Applied to a data pipeline platform, the split looks like this:

• Control plane — pipeline definitions, schedule and trigger logic, connection and credential references, role-based access rules, execution history, and monitoring and alerting.
• Compute plane — the engine that actually reads source data, executes joins and transformations, and writes results out: a Spark cluster, a warehouse's own compute, a serverless function, or a single VM.

Nothing in the first list requires running on the same infrastructure as the second. A platform can hold pipeline definitions and schedules centrally while routing the workload itself to whichever engine fits the job.

Why They're Often Coupled

Most commercial pipeline tools do not separate the two planes, and the reason has more to do with product strategy than data architecture. Shipping a single integrated runtime is easier to build, price, and support. One deployment, one support contract, one place logs live. For a vendor, that is a simpler product to sell. For a small team with no infrastructure to manage, it can be a reasonable trade to accept.

The cost shows up later. When compute and control are the same product, switching engines means switching platforms. Running one workload on a customer-owned Spark cluster and another on a serverless option usually is not supported, because the platform was never built to separate the two decisions in the first place. Governance rules, audit logs, and scheduling logic all live inside a runtime the team does not fully control and cannot relocate without rebuilding pipelines from scratch.

This is a trade-off, not a design flaw. A tightly coupled platform removes decisions a team may not want to make day to day. It just removes them permanently, not temporarily.

What Decoupling Changes

Separating the planes changes four things concretely, and none of them is free.

• Portability. A pipeline defined once can run against different compute engines without a rewrite, provided the control layer abstracts engine-specific execution details. That mirrors the portability goal NIST describes in its cloud computing reference architecture: the ability to move a workload between environments without being restricted to one provider's proprietary implementation.^[4]
• Infrastructure ownership. Compute can run inside a vendor's environment or inside the customer's own cloud account or data center. That decision determines who provisions capacity, who pays the compute bill directly, and who gets paged when a cluster runs out of memory.
• Security boundaries. When compute runs inside the customer's own network, the data being processed never has to leave it. Databricks' classic compute plane works this way: cluster VMs and the data they touch stay inside the customer's cloud account, while the control plane holds only metadata and orchestration state.^[2] A tightly coupled platform generally cannot offer that boundary, because compute and control share the same infrastructure by design.
• Cost allocation. Compute spend becomes visible and attributable to whichever infrastructure actually ran the job, instead of folded into a single platform fee that obscures which pipeline is expensive and why.

None of this means decoupling is inherently cheaper or automatically reduces lock-in. A team can still standardize on a single compute engine after decoupling, and vendor dependency simply shifts from the orchestration layer to the compute layer instead of disappearing. What decoupling buys is the option to make that choice deliberately, pipeline by pipeline, instead of inheriting it from the platform.

Architecture Patterns

In practice, most decoupled deployments land on one of four patterns, and a single organization often runs more than one at once.

None of these is universally correct. The pattern that fits depends on the workload: a warehouse-native transformation on data that already lives there is a poor candidate for a separate Spark cluster, while a large join across systems the warehouse does not touch usually is not.

Operational Risks of Decoupling

Decoupling is not free of cost, only free of a specific kind of lock-in. What it costs is integration surface.

Each supported engine becomes a separate compatibility contract. A change to how Databricks handles cluster authentication, or a new Spark version with different default behavior, is now something the control layer has to track in addition to its own release cycle. Airflow's own architecture shows a version of this problem: a scheduler that supports multiple executors, Local, Celery, Kubernetes, has to maintain a common interface across all of them so switching executors does not change how a DAG behaves.^[3] A control plane supporting multiple compute engines carries the same obligation.

Observability gets harder to keep consistent. Logs, metrics, and failure states look different depending on which engine ran the job. A control plane that presents a single monitoring view has to normalize formats across engines it does not directly operate, or the dashboard quietly becomes less trustworthy for whichever engine gets the least attention.

Credential and network management multiply. One managed runtime means one set of service credentials and one network path to secure. Three supported engines can mean three sets of credentials, three network boundaries, and three places a misconfiguration can go unnoticed until it fails in production.

Decision Checklist

Before deciding whether to decouple, or how far, answer these questions for the pipelines actually in scope. The answers rarely point the same direction for every workload in an organization.

DataFuseAI Architecture Example

DataFuseAI separates these responsibilities in practice. Pipeline definitions, scheduling, role-based access controls, execution audit logs, and the unified monitoring dashboard sit in a control layer that stays constant regardless of which engine a given pipeline runs on. Compute engine selection is a separate, per-pipeline decision: a workload can run on Databricks, on Apache Livy, or on DataFuseAI's own native engine, without changing how that pipeline is scheduled, governed, or monitored.

Deployment model reinforces the same separation. DataFuseAI ships in three GA configurations: managed cloud SaaS, private-hosted, and on-premise offline. In the on-premise offline model, both the control layer and the compute engine run entirely inside the customer's own network with no external internet connectivity required, which collapses the boundary between the two planes down to zero external exposure. In the cloud SaaS and private-hosted models, the same control layer, pipeline definitions, scheduling, governance, and monitoring, stays consistent while the underlying compute footprint changes.

When Tightly Coupled Architecture Is Better

Decoupling is not the correct default for every team. A small team without dedicated infrastructure staff often gets more value from a single vendor-managed runtime than from the flexibility to choose engines per pipeline.

A fully managed, tightly coupled platform removes real operational weight: no cluster sizing decisions, no separate credential rotation schedule for a second engine, no compatibility testing when the vendor ships an update. For a team running a handful of pipelines with predictable volume, that trade is often the right one. The cost of decoupling, additional integration surface and more infrastructure to reason about, tends to pay off only once the number of pipelines, the diversity of workloads, or the data residency requirements make engine choice worth deciding pipeline by pipeline. The risks nobody names around vendor lock-in apply here too: coupling isn't inherently risky, it's a bet on one vendor's roadmap matching your needs indefinitely.

The honest version of this decision is not "decoupled architecture is better." It is: centralize what benefits from being centralized, and keep the option to decouple compute open for the pipelines where it actually matters.

Frequently Asked Questions