ETL Proof of Concept: A Five-Phase Evaluation Framework

Why Most ETL PoCs Produce the Wrong Answer

The standard ETL evaluation goes: request demos from three or four vendors, compare connector counts, run a sample pipeline on clean data, assess the interface, check pricing. This process establishes something — that the tool can demonstrate its own capabilities in controlled conditions. What it doesn't establish is whether the tool works in yours.

Vendor demos share a structural characteristic: they're built to succeed. The dataset is pre-cleaned. The sources are the ones the platform handles most naturally. The transformation logic is simple and well-documented. The hardware running the demo is not your hardware. When you're evaluating no-code ETL tools, demo performance tells you something about the product's upper bound — not about its behavior on your specific workload, in your specific environment, with your specific constraints.

The discovery failures follow a pattern. A team spends three weeks comparing UI workflows across four platforms. They build mock pipelines, attend multiple demo sessions, run pricing models, and prepare a shortlist. Then they learn that three of the four finalists are managed cloud SaaS with no path to private-hosted deployment — a constraint that, for any organization under data residency requirements or sovereignty mandates, should have been a first-filter. The evaluation work on those three platforms was entirely wasted. The right constraint was the cheapest and fastest to verify, and it was tested last.

The governance version of this failure is slower and more expensive. A platform with "role-based access control" and "audit logging" passes initial evaluation. Twelve months into production, a compliance review surfaces that the audit logs don't capture configuration state at execution time — only that a pipeline ran, not what parameters it ran with. In January 2025, the SEC issued combined civil penalties of $63.1 million against twelve financial firms for recordkeeping failures, where the distinguishing factor between a $600,000 penalty and a $12 million penalty was the presence or absence of runtime-generated records.^[1] Governance depth that appears equivalent in a demo diverges materially when an auditor arrives.

A well-designed ETL proof of concept answers five specific questions, in this order:

1. Does the tool run in our environment? (Deployment constraint)
2. Does it handle our actual data across our actual sources and destinations? (Real data testing)
3. Does its governance model meet our compliance obligations at the operation level? (Governance depth)
4. What is its processing ceiling at our data volumes on our hardware? (Scale threshold)
5. Does the evidence generated support a confident, documented decision? (Documentation and sign-off)

Standard PoCs address question two and occasionally four. This framework addresses all five — starting with the one that costs the least to answer and eliminates the most options.

What a Well-Sequenced ETL PoC Tests (and in What Order)

Sequence is the entire argument. Running phases in the wrong order doesn't just waste time — it produces overconfidence in tools that will fail on the criteria you haven't tested yet. The framework is designed so that each phase eliminates vendors before the next phase requires investment in them. The cheapest tests come first. The most expensive ones come after the filters.

The five phases:

1. Deployment Constraint Filter. Verify deployment model compatibility before testing a single connector. This phase eliminates vendors in hours, not days — and for teams with data residency requirements, regulatory mandates, or zero-egress obligations, it eliminates the majority of the market immediately. Three deployment models exist. Most tools in this category support only one.
2. Real Data Testing. Run your actual source data through your actual transformation patterns to your actual destination. Not vendor demo data. Not a simplified sample. The pipeline that matters is the one that processes your schema, handles your specific edge cases, and writes to your system.
3. Governance Depth Verification. Confirm what the platform's governance model actually enforces at the operation level — not whether the vendor demo includes a settings panel with role labels. This phase distinguishes tools that pass compliance reviews from tools that surface gaps during them.
4. Scale Threshold Testing. Establish where performance degrades and at what volume. Every vendor claims enterprise-grade scale. The PoC determines what that means for your data volumes, your hardware configuration, and your acceptable processing windows.
5. Documentation and Decision. Convert PoC findings into a decision-ready record — written evidence of deployment verification, governance depth, scale ceiling, and connector coverage gaps per vendor. A PoC that doesn't produce this document generated experience, not evidence.

Each phase gates the next. If Phase 1 eliminates a vendor, no further testing is needed for that vendor. If Phase 2 reveals fundamental connector incompatibility, Phases 3 and 4 for that vendor are skipped. This sequencing — cheapest tests first, most constraint-revealing tests early — is exactly what the standard evaluation process inverts.

Phase 1 — Deployment Constraint Filter

Deployment model compatibility is binary. Either the tool runs in your environment or it doesn't. Verifying this requires no pipeline builds, no connector configuration, and no vendor-customized sandbox — only a direct review of deployment documentation and two or three specific questions to the vendor's sales team. It should take half a day per vendor. Most teams run it after weeks of evaluation. The order should be reversed.

Three deployment models exist across the ETL tool market:

The vendor hosts and manages everything. Data moves through vendor infrastructure. For teams without data residency requirements, regulatory mandates around sovereignty, or zero-egress obligations, this is entirely acceptable. For teams with any of those constraints, it's a disqualifying characteristic — and it's the only model the majority of tools in this category support.

The platform runs on the organization's own infrastructure, with vendor-assisted setup. Data stays within the organization's control. This satisfies most data residency requirements and provides deployment options for internally managed environments. Private-hosted does not mean fully offline — most private-hosted options still require internet connectivity for license validation, connector updates, or telemetry.

The platform runs with zero external internet connectivity, entirely within the customer's private network. This is a hard requirement for government contractors, defense-adjacent teams, and healthcare organizations with zero-egress mandates. It is rare. Many platforms that describe themselves as "supporting on-premise deployment" actually mean private-hosted with internet connectivity. The distinction matters and must be verified explicitly.

Phase 1 verification steps: Request deployment documentation, not a demo — documentation specifies what's GA; demos almost always run on managed cloud. Ask specifically: "Is your on-premise deployment available with zero external internet connectivity, and is that configuration currently generally available?" Ask: "Does your private-hosted option require internet connectivity for license validation, telemetry, or connector updates?" Confirm the deployment model you need is available at the pricing tier you're evaluating — some deployment options exist only at enterprise pricing tiers not yet in scope. Document the result per vendor: in scope or eliminated.

Phase 2 — Real Data Testing

A vendor demo proves the tool works on their data. A PoC proves it works on yours.

The distinction is more significant than it sounds. Vendor demos use clean, pre-optimized datasets from controlled environments. They don't include the PostgreSQL instance with the unconventional schema that represents your most critical source. They don't include the Oracle database running a version where connector behavior has edge cases. They don't include the transformation logic that handles your specific billing rules, the business logic that actually matters when the pipeline runs in production.

What to include in Phase 2:

Source systems. Connect to your two or three highest-priority actual source systems — not the simplest ones to configure, the most important ones to your operations. If the majority of your data originates from MySQL and a specific AWS RDS variant, those are your test sources. A connector that doesn't handle your schema reliably isn't a connector for your use case, regardless of the platform's aggregate connector count. Review the platform's data connector coverage for your specific database types before investing time in Phase 2.

Destination systems. Test against your actual destination. If you're writing to Snowflake, verify ELT pipeline behavior at your schema complexity. If you're writing to an on-premise PostgreSQL database, verify write throughput and schema handling in that environment specifically. Destination-side incompatibilities often don't appear in demo flows because demo flows use destinations the vendor has optimized for.

Transformation patterns. Build two or three transformations representative of your actual workload: a straightforward filter and aggregation, a join across two source tables, and one transformation that represents your business-specific logic. The goal isn't to build a production pipeline during the PoC — it's to verify that your transformation patterns are supported without requiring custom code workarounds.

Edge cases. Deliberately include: a source table with nulls in fields that should be non-nullable; a batch containing duplicate records requiring deduplication; a schema where a column type doesn't match the destination's expected type. Edge cases are where real pipeline failures originate in production. Testing them in the PoC is the difference between discovering a failure mode before commitment and discovering it after.

Phase 2 verification output: For each source/destination combination tested, document: connector connected successfully (yes/no), transformation executed as configured (yes/no), edge cases handled as expected (yes/no/partial), custom driver development required (yes/no), approximate throughput on your hardware in rows per second. This record is direct input to Phase 5 documentation.

Phase 3 — Governance Depth Verification

Most comparison tables mark "Has RBAC" and "Has audit logging" with a checkmark. The more important question is what those properties actually enforce. Role-label systems assign broad access categories — admin, editor, viewer — and record that a pipeline ran. Action-level systems scope permissions to specific operations: running a query, starting a cluster, editing a pipeline configuration, deleting a log entry. In a compliance context, those are not equivalent capabilities.

The audit logging distinction is equally specific. GDPR Article 32 requires controllers and processors to implement appropriate technical measures to ensure ongoing confidentiality, integrity, and availability of processing systems.^[3] HIPAA 45 C.F.R. §164.312(b) requires hardware, software, and procedural mechanisms to record and examine activity in systems containing ePHI.^[4] NIST SP 800-53 Rev. 5 AU-3 specifies that audit records must contain event type, date and time, user identity, and outcome.^[5] "Has audit logging" does not satisfy those standards. Capturing user, action, and configuration state at execution time does.

Most vendors don't distinguish between role-label and action-level governance in their marketing materials. The PoC is where the distinction becomes visible.

Phase 3 verification steps:

Ask the vendor: "Can permissions be scoped so a user can run pipelines but not edit their configuration, separately from the ability to delete log entries?" If the answer describes module-level roles — admin can do everything, editor can do most things — that's role-label access. If the answer describes specific operation-level controls within modules, that's action-level. Request documentation, not a demo answer.

Run a test pipeline — any transformation on sample data. Retrieve the execution log for that run. Verify the log contains: triggering user, run start timestamp, run end timestamp, configuration state or job version active at execution time, input record count, output record count, and success or failure status. If any of those fields are absent, the platform's audit logging doesn't satisfy NIST AU-3 content requirements. For teams with data governance and compliance obligations, this isn't a minor gap — it's the specific record an auditor will request. See also: building audit-ready data operations.

Request a structured export of the last 30 days of execution logs. The export should be machine-readable — CSV or JSON — not a summary PDF or a rendered dashboard screenshot. If the platform cannot produce a structured log export on demand, it cannot respond to formal audit requests. A regulator asking for execution records is not asking for a screenshot.

Phase 4 — Scale Threshold Testing

Every vendor claims enterprise-grade scale. The PoC determines what that means for your data volumes, on your hardware, within your acceptable processing window. Scale testing has two goals: establish where processing time starts growing non-linearly relative to data volume, and verify that the growth curve stays within your operational window — the time available for batch jobs to complete before downstream systems need the data.

How to structure scale testing: Run the same pipeline at three to four distinct volume tiers using your actual data or a proportionally representative sample. Use volumes that bracket your current production load and your expected load in 18 months — not the vendor's benchmark dataset. Record total time, read phase time, transform phase time, write phase time, and success or failure status at each tier. The time breakdown reveals where constraints live. If write time dominates at scale, the bottleneck is in your destination database's ingestion capacity. If transform time dominates, the platform's processing engine is the constraint. These require different responses.

What sub-linear scaling looks like in practice: if data volume grows 10× and processing time grows 2–3×, the platform is scaling efficiently. If time grows proportionally to data, you're at linear scaling — manageable but more expensive to sustain as volumes increase. If time grows faster than data, you've found a constraint that will compound as your operations grow.

What to ask vendors without published benchmarks: "Can you point to publicly documented benchmark results — a published test with specific volume, hardware configuration, and throughput figures?" If no such document exists, the scale ceiling is unverified. That isn't automatically disqualifying — if scale isn't a primary constraint, it's a lower-weight criterion. But it should be stated as unverified in your Phase 5 documentation, not assumed equivalent to tools that have published evidence. The Phase 4 section of your PoC record should read: tested at [volume] on [hardware], throughput [N] rows/sec, growth from baseline [X×]. That's a specification. "Handles enterprise scale" is not.

Phase 5 — PoC Documentation and Sign-Off

The purpose of Phase 5 is not to summarize what happened — it's to produce a written record that makes the decision obvious, or clearly surfaces where it isn't obvious yet. Decision-makers who weren't in the PoC room need this document to be self-contained. Engineers who weren't involved in the evaluation need it to contain specific technical findings, not impressions.

PoC documentation structure (five sections):

Section 1 — Deployment verification. For each evaluated vendor: deployment model requested, deployment model tested, whether the requested model is currently GA or in preview, deployment documentation URL. Result: in scope or eliminated. If eliminated, state why in one sentence.

Section 2 — Real data test results. For each vendor that passed Phase 1: sources tested, destinations tested, transformation patterns validated, edge cases handled (pass/partial/fail), custom development required (yes/no), throughput observed on Phase 2 hardware. Any unexpected behavior documented with specific reproduction steps.

Section 3 — Governance depth findings. For each vendor that passed Phases 1–2: RBAC granularity (role-label or action-level, confirmed from documentation), audit log fields captured per execution (list each field present), log export capability (structured/unstructured/unavailable), compliance gap identified (none, or specific gap named with regulatory mapping). This section is what a compliance officer or external auditor will read.

Section 4 — Scale ceiling. For each vendor that passed Phases 1–3: volume tiers tested, hardware configuration used, throughput at each tier, growth curve pattern (sub-linear/linear/supra-linear), operational window compatibility at expected 18-month volume. If scale was not tested, state explicitly that this criterion is unverified for this vendor.

Section 5 — Decision recommendation. Two to four sentences: which vendor meets all hard constraints, which criteria were determinative, what risks (if any) the selected vendor carries that require mitigation planning. If two vendors passed all phases with genuinely different tradeoff profiles, state the decision explicitly: the PoC has surfaced a real choice between [X tradeoff] and [Y tradeoff], and the organization's priority between those two factors determines the outcome.

The document's job is to make the decision obvious — or to clearly surface where it isn't yet. Genuine ambiguity, documented precisely, is a more valuable PoC output than false confidence from an evaluation that skipped three of the five phases.

On PoC duration: Two to three weeks for a structured evaluation of two to three vendors, with one data engineer at part-time commitment. Phase 1 takes half a day per vendor. Phases 2 through 4 take three to five days per vendor when testing with real data and real infrastructure. Phase 5 takes one day. Teams compressing this to a single week typically cut Phase 3 — governance depth verification — and encounter the gap later, at a point where the cost of discovering it is significantly higher. The build-vs-buy decision for data pipelines carries similar sequencing logic: the cheapest discovery is always the earliest one.

Frequently Asked Questions