ETL Advisors | Loading Databricks from Talend: Delta Lake, Unity Catalog, and MERGE That Reruns Cleanly

Databricks now shows up on our engagements about as often as Snowflake, and almost never as a greenfield build. The usual shape: the analytics team has standardized on Databricks SQL and Unity Catalog, and a few hundred Talend jobs still carry a decade of extraction and business logic aimed at an on-prem warehouse. Nobody is funding a rewrite of those jobs. They need a new landing zone and a load pattern that survives reruns.

The good news, as with Microsoft Fabric and Iceberg, is that this needs no exotic component set. Databricks reads Parquet from cloud object storage and speaks SQL over JDBC. Talend already does both well. What actually decides whether the project goes smoothly is where you put the boundary between "Talend moves bytes" and "Databricks does set-based SQL".

Step 1: Draw the boundary before you build anything

The pattern that holds up in production is deliberately boring:

Talend extracts, cleanses, and writes Parquet files to a staging path in S3 / ADLS Gen2 / GCS.
Talend issues SQL statements over JDBC — COPY INTO, then MERGE — and lets the Databricks cluster do the heavy set-based work.
Talend owns orchestration, logging, and the audit trail.

The anti-pattern is streaming millions of rows through a tDBOutput on a JDBC connection and doing row-level upserts. It will work in a smoke test with 10,000 rows and then take six hours on the real volume. Delta tables want bulk writes and set-based merges; JDBC row inserts fight that design. This is the same pushdown argument we make in Talend job performance tuning, and it matters more here than almost anywhere else.

Step 2: Get authentication right once

There are two separate credentials in play and teams routinely conflate them.

Storage credentials let Talend write staged files. Use the cloud-native mechanism — an IAM role for the runtime host on AWS, a service principal or managed identity on Azure. Keep them in context variables loaded from your vault or via implicit context load, never in a committed context group (our notes on multi-environment context handling).

Databricks credentials let Talend run SQL. Use a service principal with an OAuth M2M secret, not a personal access token belonging to whichever consultant built the job. PATs die when the human leaves, and they inherit that human's permissions, which is usually far too much. In the JDBC URL, OAuth M2M looks like:

jdbc:databricks://adb-<workspace-id>.<n>.azuredatabricks.net:443/default;
  httpPath=/sql/1.0/warehouses/<warehouse-id>;
  AuthMech=11;Auth_Flow=1;
  OAuth2ClientId=<client-id>;OAuth2Secret=<secret>

Drop the DatabricksJDBC42.jar driver into your job's library set and use a generic JDBC connection if your Talend version predates a dedicated Databricks component. Point httpPath at a SQL warehouse, not an all-purpose cluster — warehouses start in seconds and auto-stop, which matters for the cost discussion below.

Step 3: Register the staging area in Unity Catalog

This is the step that gets skipped, and then COPY INTO fails with a permission error that reads like a storage problem but is a governance problem. Under Unity Catalog, a raw s3:// or abfss:// path is not automatically readable, even if your cluster's identity can see it.

You need, once, from a workspace admin:

a storage credential wrapping the IAM role or managed identity,
an external location binding that credential to the staging prefix,
READ FILES (and WRITE FILES if Talend writes through Databricks) granted to your service principal on that location.

CREATE EXTERNAL LOCATION talend_staging
  URL 'abfss://staging@acmedata.dfs.core.windows.net/talend'
  WITH (STORAGE CREDENTIAL acme_uc_cred);

GRANT READ FILES ON EXTERNAL LOCATION talend_staging TO `talend-svc`;
GRANT USAGE ON CATALOG prod TO `talend-svc`;
GRANT USAGE, CREATE TABLE ON SCHEMA prod.raw TO `talend-svc`;

If your platform team prefers, a Unity Catalog volume (/Volumes/prod/raw/talend_staging/...) is the friendlier alternative: same governance, but the path is addressable as a managed name instead of a cloud URL. Either is fine; pick one per environment and make it a context variable so DEV, UAT and PROD differ only in configuration.

Step 4: Write partitioned Parquet, one load ID at a time

Job skeleton:

tPreJob -> tSetGlobalVar (load_id = yyyyMMddHHmmss)
tDBInput -> tMap -> tFileOutputParquet (local staging)
        |
        OnSubjobOk -> tS3Put / tAzureStoragePut  (upload)
        OnSubjobOk -> tDBRow  (COPY INTO)
        OnSubjobOk -> tDBRow  (MERGE)
tPostJob -> tDBRow (audit insert) + cleanup

Write Parquet rather than CSV unless something forces your hand. Parquet carries types, so decimal precision and timestamps stop drifting between the source and the lakehouse — the single most common root cause of "the dashboard numbers do not match" after a cutover.

Put the load ID in the path, not just the filename:

talend/orders/load_id=20260311T0210/orders-part-0001.parquet

A rerun of the same load ID then overwrites a known, bounded set of files instead of appending duplicates. This is the cheapest idempotency you will ever buy.

Step 5: COPY INTO for the append case

For an append-only landing table, COPY INTO is the right tool and it is already idempotent — Databricks tracks which files it has ingested and skips them on rerun:

COPY INTO prod.raw.orders_stg
FROM 'abfss://staging@acmedata.dfs.core.windows.net/talend/orders/load_id=20260311T0210/'
FILEFORMAT = PARQUET
COPY_OPTIONS ('mergeSchema' = 'false');

Keep mergeSchema off in production. Silent schema evolution is how a renamed source column becomes a new nullable column full of nulls and nobody notices for a quarter. If your sources genuinely drift, handle it explicitly with the contract approach in handling schema drift in Talend.

Step 6: MERGE for upserts, with dedup inside the source

For a curated table with keys, run a MERGE from the staging table. Two details make it rerun-safe:

MERGE INTO prod.core.orders AS t
USING (
  SELECT * FROM (
    SELECT *, ROW_NUMBER() OVER (
             PARTITION BY order_id ORDER BY source_updated_at DESC) AS rn
    FROM prod.raw.orders_stg
    WHERE load_id = '20260311T0210'
  ) WHERE rn = 1
) AS s
ON t.order_id = s.order_id
WHEN MATCHED AND s.source_updated_at > t.source_updated_at
  THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

First, deduplicate inside the USING subquery. Delta raises an error when multiple source rows match one target row, and a CDC batch will contain several versions of the same key — see CDC patterns that scale.

Second, guard the UPDATE with a watermark comparison. That makes a replayed batch a no-op instead of resurrecting stale values, and it keeps late-arriving out-of-order events from overwriting newer state. If you are maintaining history rather than current state, the same staging table feeds the Type 2 pattern — just target your dimension with validity dates instead of UPDATE SET *.

A note on liquid clustering: on newer runtimes, CLUSTER BY on the merge key usually beats hand-tuned partitioning for merge-heavy tables. Fewer files rewritten per merge, no partition-skew tuning.

Step 7: Fail loudly, and account for cost

tDBRow will happily swallow a SQL failure if you do not check it. Wire tDBRow into tStatCatcher/tLogCatcher and assert on affected row counts, per our error handling and observability patterns. A merge that silently updated zero rows is a failed load, not a successful one.

On cost, three habits save real money:

One warehouse per workload class, sized small, with auto-stop at a couple of minutes. Talend jobs are bursty; idle compute is pure waste.
Batch your SQL. Ten thousand single-row tDBRow calls each pay JDBC and query-planning overhead. One COPY INTO plus one MERGE per table per run is the target.
Run OPTIMIZE and VACUUM on a schedule, not inside the load job. Merge-heavy Delta tables accumulate small files; leaving that to a weekly maintenance job keeps your nightly window predictable.

Where this leaves the migration conversation

Adopting Databricks does not mean discarding your Talend estate. It means demoting Talend from "the thing that does the transformation" to "the thing that reliably lands governed, typed, idempotent batches" — and letting Delta SQL do set-based work it is much better at. That is the same division of labour we describe in Talend and dbt together and in ETL vs ELT in 2026.

If you are staring at a few hundred jobs and a Databricks mandate, get in touch — mapping the landing pattern once, before anyone touches job number two, is what keeps the other several hundred boring.