Processing Engines
DataFuseAI lets you define, manage, and scale the compute engines that power your data workflows. Create engine profiles to control where queries, pipelines, and jobs run—whether on the DataFuseAI native engine, Databricks, or an existing Spark cluster through Livy.
Engine Types in DataFuseAI
Engines define where your data work actually runs. Each engine maps to a real compute environment that DataFuseAI connects to — so orchestration stays consistent even when execution environments differ.
DataFuseAI Native Engine
RecommendedThe built-in execution environment that ships with DataFuseAI. It's a practical default for everyday batch workloads, scheduled jobs, and initial setups where standing up or managing external clusters would add unnecessary overhead. This is often where teams start — and, for many use cases, where they stay.
Databricks Engine
EnterpriseConnect DataFuseAI to an existing Databricks workspace and reuse your Databricks clusters for execution. DataFuseAI handles orchestration, scheduling, and visibility, while Databricks continues to handle scaling and compute-intensive workloads. This option makes sense when Databricks is already a core part of your data platform and you want to centralize control without disrupting it.
Apache Livy Engine
AdvancedIntegrate with Spark clusters via a Livy endpoint. Jobs are submitted and managed over HTTP, which keeps cluster internals hidden from end users while still supporting large-scale, Spark-based processing. This is typically used in environments with existing Spark infrastructure and stricter control over how clusters are accessed.
Engine Configuration Workflow
Engines are configured as named profiles that encapsulate everything needed to run workloads - connection details, authentication, and environment context.
Create Engine Profiles
From the Engine page, define a new engine by selecting the engine type, assigning a name, and providing the required connection details (such as workspace URLs, Livy endpoints, or internal engine settings).
Authentication & Credentials
Authentication is configured at the engine level and varies by engine type - tokens or service credentials for Databricks, supported auth methods for Livy. Credentials live in the engine profile, not inside queries or pipelines, so they're easier to rotate and audit.
Cluster & Workspace Details
For engines like Databricks, profiles are bound to specific workspaces and clusters. This removes ambiguity about where workloads will run and helps avoid accidental execution in the wrong environment.
Test Connection & Status
Each engine profile includes a Test Connection action, allowing you to verify connectivity and permissions before using the engine in queries, pipelines, or jobs.
How Engines Are Used in the Platform
Once configured, engines become selectable execution backends across multiple modules.
Query Execution
When running queries, users choose which engine should execute the workload, aligning compute choice with data location and query complexity.
Pipeline Execution
Pipelines use engine profiles as their compute target, so complex transformations run in the right environment without changing pipeline logic.
Job Scheduling
Scheduled jobs rely on engine profiles to ensure pipelines consistently execute in the intended environment — day after day, run after run.
Run History & Diagnostics
All executions tied to an engine contribute to shared run history and logs, making it easier to isolate engine-specific issues without hunting across systems.
Key Benefits
Start Quickly, Scale Gradually
Start with the native engine, then move heavier workloads to Databricks or Spark as requirements grow — without rec pipelines.
Consistent Orchestration Across Environments
Use different engines for dev, staging, and production — or for different teams — while keeping a single orchestration layer, unified UI, and shared operational model.
Reliable Batch & Automation Workloads
DataFuseAI is optimized for recurring ETL, scheduled transformations, migrations, and routine data processing — delivering predictable runs and easier troubleshooting across engines.
Distributed Processing Capability
Powerful distributed compute engines for large data volume. DataFuseAI coordinates execution, while platforms like Databricks and Spark scale compute horizontally behind the scenes.
FAQ
Frequently Asked Questions About Processing Engines
It is the runtime that actually executes your queries, pipelines, and jobs. DataFuseAI holds the pipeline definition — the sources, transformations, and destinations you configured — and hands the work to an engine to run. Because the two are separate, the same pipeline can run on a different engine without being rebuilt.
Three. The DataFuseAI native engine runs queries, pipelines, and jobs without any external cluster, so a new tenant can start immediately. A Databricks engine profile points at your own workspace and clusters and orchestrates the workload from inside DataFuseAI. An Apache Livy profile attaches to a Spark cluster you already run, which keeps the infrastructure under your control.
Match the engine to the workload and to what you already operate. The native engine suits work that does not justify a cluster and teams who would rather not run one. Databricks makes sense where the workspace already exists and you want DataFuseAI to orchestrate against it. Livy fits when you run your own Spark cluster and need the compute to stay in your environment. The choice is per profile, not per platform, so more than one can be in use at once.
A saved connection to a runtime: the workspace or cluster details, the credentials, and the settings that describe how work reaches it. Registering it once means pipelines and jobs reference the profile rather than repeating connection details, and changing where work executes becomes a change to the profile rather than an edit to every pipeline that uses it.
Every engine profile includes a Test Connection action. It checks connectivity and permissions against the target runtime, so a wrong credential or an unreachable cluster surfaces at configuration time rather than in the middle of a scheduled job. Running it after any change to workspace details or credentials is the cheapest way to keep a nightly run from failing silently.
Yes — maintain a separate engine profile for each. Experiments then execute against the development runtime while scheduled production jobs continue on their own, so a test cannot consume the compute a critical job depends on. Because pipelines reference the profile, promoting work from development to production is a matter of pointing it at the other profile.
No. The native engine runs queries, pipelines, and jobs without any external cluster, which is what a new tenant uses by default. Databricks and Livy profiles exist for teams that already operate that infrastructure and want the work to run there — they are an option for reusing what you have, not a prerequisite for using the platform.
Yes, because the pipeline definition and the runtime are separate concerns. The sources, transformations, and destinations you configured stay as they are, and the engine profile determines where that work executes — so a job can be moved onto a larger runtime as volume grows without the pipeline being rebuilt.
