Platform Architecture
DataFuseAI is organized into clear modules — Engines, Drivers, Connection Profiles, Queries, Pipelines, Jobs, File Explorer, and Tenant/User/Group management. The structure is intentional. The way the platform works internally mirrors what users see in the UI, so there's less mental translation between 'how it's built' and 'how it's used.'
Core Platform Modules
Each core function in the UI maps to a module in the architecture, making behaviors predictable and transparent.
Dashboard & Monitoring
The Dashboard aggregate key operational metrics in one place: job and pipeline activity, query usage, connection status, recent actions, and overall infrastructure system state for each tenant. It's designed to answer the first question operators usually ask: is everything running as expected?
Engines, Drivers & Connection Profiles
These three modules work together to separate compute, connectivity, and credentials. Engines define where workloads run, drivers define how systems are accessed, and connection profiles bind those drivers to real databases, storage systems, and file locations.
Queries, Pipelines & Jobs
Queries allow direct interaction with data. Pipelines define repeatable flows and transformations. Jobs determine when and how those pipelines run. Together, they form a clear progression from exploration to automation.
File Explorer & Artifacts
File Explorer provides a structured way to organize pipelines, jobs, queries, and connection profiles. Assets are grouped under user and system folders, making it easier to manage ownership, sharing, and cleanup over time.
Data & Workflow Lifecycle
The architecture follows a consistent lifecycle: connect → transform → schedule → monitor.
Connect with Drivers & Profiles
Drivers provide support for specific platforms—relational databases, NoSQL systems, object storage, and documents. Connection profiles handle the practical details: hostnames, ports, credentials, and database configuration, keeping those concerns reusable and secured.
Transform with Pipelines
Pipelines use visual, drag-and-drop nodes to reshape data as it moves between sources and targets. Common operations—aggregate, dedupe, filter, join, union, split, route, derived fields, explode, and window functions—are first-class, making transformations explicit and inspectable.
Automate with Jobs
Jobs orchestrate one or more pipelines on a cron schedule. This is where ad-hoc work turns into repeatable processes, supporting routine ETL, batch processing, and recurring data movement.
Observe with Dashboards & Logs
Execution history, success and failure counts, recent activity, and health indicators roll up into dashboards and logs. The goal isn't just visibility, but faster understanding when something changes or breaks.
Security, Tenancy & Governance
Access and actions are controlled at multiple levels, with detailed permissions tied to each module.
Tenant & User Model
Each tenant operates as an isolated workspace with its own users, groups, and configuration. A SuperAdmin manages company-level settings, user access, and module visibility.
Fine-Grained Permissions
Permissions are action-specific—running queries, starting or stopping clusters, creating drivers, editing pipelines, deleting logs, managing jobs. This allows roles to be defined precisely, without granting broad access by default.
Isolated Access to Data
Engines, drivers, connection profiles, and pipelines are all governed by permissions. Sensitive data stays scoped to the roles that need it, rather than leaking through shared infrastructure.
Auditability & History
User actions and system operations are logged with context, making it possible to review who ran what, when, and with which configuration—supporting audits, reviews, and compliance workflows.
Key Benefits
Why teams choose DataFuseAI when efficiency, governance, and operational reliability matter.
Aligned with the UI
The architecture follows the same conceptual model users see in the interface, reducing cognitive load and making onboarding easier.
Governance by Design
RBAC, isolation, and action-level permissions are built into the core architecture, not added later as an afterthought.
Reliable Automation Path
From connection profiles to jobs, the architecture is optimized for repeatable, stable, and monitored batch workflows.
Built to Grow
You can add engines, drivers, pipelines, and tenants over time without changing how operators or users think about the system. The mental model stays stable, even as scale increases.
FAQ
Frequently Asked Questions About Platform Architecture
Four groups that follow the order the work happens in. Engines, drivers, and connection profiles define what the platform can reach and where work executes. Queries, pipelines, and jobs are the work itself — exploring data, transforming it, and running that transformation on a schedule. File explorer and artifacts hold the files a workflow produces or consumes. Dashboards and logs report what ran and what happened.
Around the same conceptual model you see in the interface, deliberately. A connection profile in the architecture is the connection profile on screen; a job is a job. Keeping the internal structure and the user-facing vocabulary aligned is what makes the platform learnable — a new team member does not have to hold two different mental models to be productive.
Each tenant operates as an isolated workspace with its own users, groups, and configuration, so one tenant's pipelines, profiles, and files are not visible to another. A SuperAdmin manages company-level settings, user access, and which modules are visible. Isolation is a property of the architecture rather than a setting applied afterwards.
With role-based access control, workspace isolation, and action-level permissions built into the core architecture rather than layered on later. Permissions are assigned to groups, so access follows a role rather than an individual, and what a user may do — run, edit, view — is distinguished from what they may see. That distinction is what lets an analyst execute a pipeline without being able to change it.
Connect, build, run, observe. You connect with drivers and connection profiles; build with queries and pipelines; run them as jobs on a schedule; and observe the result through dashboards and logs. The lifecycle is deliberately repeatable — the same pipeline produces the same shape of output every run, which is what makes the output trustworthy enough to report on.
Batch. From connection profiles through to jobs, the architecture is optimised for repeatable, stable, monitored batch workflows — scheduled runs with recorded history rather than continuous event processing. That focus is why run visibility and reproducibility are first-class: a batch pipeline is judged on whether last night's run can be explained.
Wherever the engine profile points. Execution is configured separately from the platform itself, so work can run on the DataFuseAI native engine, an existing Databricks workspace, or a Spark cluster reached through Apache Livy. Because the two are decoupled, moving a workload onto larger compute does not require the pipeline to be rebuilt.
Yes. The architecture supports managed, private-hosted, and on-premise deployment, with the same platform features in each. On-premise runs locally with no external connectivity requirement, which is the model organizations choose when internal policy or a regulator requires the environment and the data path to stay inside their own perimeter.
