Writing to Apache Iceberg from Talend: Lakehouse Patterns That Hold Up

Every lakehouse conversation in 2026 ends up at the same place: the table format. Snowflake, Databricks, BigQuery, Redshift, Trino, and Athena all read and write Apache Iceberg now, and platform teams are landing raw and curated data in Iceberg tables on S3 or ADLS instead of in a single warehouse.

Talend Studio has no tIcebergOutput component, and it probably never will. That surprises people into thinking Talend cannot participate in a lakehouse. It can, and on migration projects it is usually the cheapest option available — you just write to Iceberg through an engine that already speaks Iceberg, instead of writing Iceberg files yourself.

This tutorial covers the three patterns we actually deploy, the MERGE logic that keeps Iceberg tables idempotent, how to survive schema evolution, and the maintenance jobs nobody remembers until query times double.

First, the rule that saves you a rewrite

Do not try to write Parquet plus manifest metadata from Talend directly. It is technically possible with a Java routine and the Iceberg Java API, and we have seen exactly one shop attempt it. Iceberg's snapshot, manifest-list, and manifest-file layout is a moving target across format versions (v2 versus v3 deletion vectors, for example), and a partially written commit is a corrupt table. Let a real Iceberg writer own the commit.

That leaves Talend doing what Talend is good at: extraction, complex row-level transformation, orchestration, and error handling. The commit is delegated.

Pattern 1: Staged files plus engine-side COPY/MERGE (the default)

This is the pattern we recommend first for at least 80% of cases.

1. Talend extracts and transforms, then writes partition-aligned files to object storage with tFileOutputParquet (or tFileOutputDelimited if you must — Parquet is worth the extra effort).
2. Talend uploads with tS3Put / tAzureStoragePut into a landing prefix that includes a batch key: s3://lake-landing/orders/batch_dt=2026-02-11/run_1873/.
3. Talend calls the engine with tDBRow to register the files and MERGE them into the Iceberg table.

The extract-and-write half looks like any other Talend job. The interesting part is step 3. With Snowflake managing Iceberg tables, or with Spark SQL / Trino over a REST catalog, the SQL is roughly the same shape:

-- Register the batch as a temporary external view (Trino/Spark flavour)
CREATE OR REPLACE TEMPORARY VIEW stg_orders
USING parquet
OPTIONS (path 's3://lake-landing/orders/batch_dt=2026-02-11/run_1873/');

MERGE INTO lake.curated.orders  AS t
USING (
  SELECT * FROM (
    SELECT s.*,
           ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY updated_at DESC) AS rn
    FROM stg_orders s
  ) WHERE rn = 1
) AS s
ON  t.order_id = s.order_id
WHEN MATCHED AND s.op = 'D' THEN DELETE
WHEN MATCHED AND s.updated_at > t.updated_at THEN UPDATE SET *
WHEN NOT MATCHED AND s.op <> 'D' THEN INSERT *;

Three details in that statement do all the work:

• The ROW_NUMBER() de-duplication. Iceberg MERGE fails or produces non-deterministic results when the source contains multiple rows for the same key. CDC batches almost always do. Collapse to the latest row per key inside the subquery, every time.
• The updated_at > guard on UPDATE. This makes the MERGE safe to re-run. Replay the same batch and no rows change, so a failed job can simply be restarted rather than surgically repaired.
• Explicit delete handling. If your source emits soft deletes, decide once whether the curated table hard-deletes or carries an is_deleted flag, and document it. Mixed conventions across tables are a permanent tax on every downstream consumer.

Parameterise the batch path with a context variable so the SQL is built once:

"MERGE INTO lake.curated." + context.table_name + " AS t USING ( ... 's3://" 
  + context.landing_bucket + "/" + context.table_name + "/batch_dt=" 
  + TalendDate.formatDate("yyyy-MM-dd", TalendDate.getCurrentDate()) 
  + "/run_" + context.run_id + "/' ... )"

Keep the SQL in a file or a routine rather than inline in tDBRow, so it is reviewable in Git. Long inline SQL in a component parameter is invisible to code review and to grep.

Pattern 2: Warehouse-managed Iceberg tables

If your platform is Snowflake or BigQuery and the warehouse can be the Iceberg catalog owner, this is the lowest-effort route: Talend loads a normal staging table exactly as it always has (see our Snowflake loading tutorial for the bulk/stage mechanics), and a tDBRow step MERGEs staging into an Iceberg-format target table.

The Iceberg-ness becomes an attribute of the target DDL rather than something your job knows about:

CREATE ICEBERG TABLE curated.orders (...)
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = 'lake_vol'
  BASE_LOCATION = 'curated/orders/';

Your Talend job does not change at all. That is the point — and it is why we usually push clients here before they try anything more exotic. External engines read the table through the catalog; the warehouse handles commits, compaction, and snapshot expiry.

Trade-off: you are tied to that warehouse's write path, and cross-engine writes (Spark writing the same table) are restricted. For a single writer, many readers — the common case — it is fine.

Pattern 3: Talend orchestrating Spark or a REST catalog job

When transformation volume is genuinely large, or the target is a vendor-neutral catalog (Polaris, Nessie, Glue via REST), keep Talend as the orchestrator and let Spark own the write:

• tSystem or tSSH submits a spark-submit job, or
• tRESTClient calls a Databricks Jobs / EMR Serverless / Snowpark endpoint, then polls for terminal state.

The job that matters here is the polling loop, and it is where most implementations are sloppy. A pattern that behaves:

1. tRESTClient submits, captures the run ID into a context variable.
2. tLoop with a bounded iteration count wraps a tSleep plus a status tRESTClient call.
3. tJavaRow parses state; tDie on FAILED/CANCELLED; break the loop on SUCCEEDED.
4. If the loop exhausts, tDie with a message including the run ID and the console URL.

Two things that will bite you: submitting is not the same as succeeding (a job that only checks HTTP 200 on submit will report green while the Spark job dies), and unbounded polling loops hang schedulers forever. Always bound the loop, and always fail loudly with the remote run ID in the message so an on-call engineer can find the Spark log in one click.

Schema evolution without breaking readers

Iceberg tracks columns by ID, not position, which makes it far more forgiving than Hive-style tables. Adds, renames, reorders, and safe type widenings are metadata-only operations. That does not make evolution automatic, though — your Talend job still has a fixed schema baked into the component metadata.

What works in practice:

• Add columns at the target first, deploy the job second. Iceberg fills missing columns with NULL for old data, so an ALTER TABLE ... ADD COLUMN is safe to run ahead of the job release.
• Never reuse a column name for a different meaning. Drop and add with a new name; readers relying on the old ID will not silently get garbage.
• Use UPDATE SET * / INSERT * in MERGE only when the staging schema is generated from the target. Otherwise a new source column silently changes behaviour on the next deploy. On stable pipelines we prefer explicit column lists, generated into the SQL file by a small script.
• Detect drift instead of discovering it. A short tDBInput job comparing information_schema columns against a checked-in expected list, run nightly, turns "the load broke" into "the source added three columns yesterday."

The maintenance jobs everyone forgets

Iceberg tables written in micro-batches accumulate small files and snapshots. Query performance degrades quietly for weeks, then someone declares "the lakehouse is slow." Schedule maintenance as ordinary Talend jobs — they are just SQL calls, and putting them in Talend means they inherit your existing logging and alerting:

-- compaction (Spark/Trino syntax varies by engine)
CALL system.rewrite_data_files(table => 'lake.curated.orders');
CALL system.rewrite_manifests(table => 'lake.curated.orders');
-- retention: keep 7 days of snapshots for time travel
CALL system.expire_snapshots(table => 'lake.curated.orders', older_than => TIMESTAMP '2026-02-04 00:00:00');
CALL system.remove_orphan_files(table => 'lake.curated.orders');

Weekly compaction and daily snapshot expiry is a reasonable starting point for tables loaded hourly. Run remove_orphan_files less often and never with a lookback shorter than your longest-running write job, or you will delete files belonging to an in-flight commit.

Also: keep a record of the snapshot ID after each load. SELECT snapshot_id FROM lake.curated.orders.snapshots ORDER BY committed_at DESC LIMIT 1 into a control table gives you a one-line rollback (CALL system.rollback_to_snapshot(...)) when a bad batch lands. That is the single most valuable Iceberg feature for ETL teams, and it costs one tDBInput and one insert per run.

What to check before you commit to any of this

• Does your catalog support the writes you need? Glue, Polaris, Nessie, and warehouse-managed catalogs differ meaningfully on concurrent-writer support.
• Is there exactly one writer per table? Multi-writer Iceberg works, but optimistic concurrency means retries, and your MERGE must be replay-safe (see the updated_at guard above).
• Are your file sizes sane? Aim for 128–512 MB target files. Talend jobs that write a file per 5,000 rows will hand you a small-file problem before the first month is out.
• Can you prove idempotency? Re-run yesterday's batch into a clone (CREATE TABLE ... CLONE) and diff. If the second run changes rows, your MERGE is not safe and your restart procedure is a rewrite.

None of this requires abandoning Talend. It requires drawing the boundary in the right place: Talend extracts, transforms, orchestrates, and reports; the engine commits. Teams that respect that split get Iceberg tables in a couple of sprints. Teams that try to make Talend a table-format writer spend a quarter learning why nobody does that.

If you are planning a lakehouse migration and want a second opinion on where the boundary should sit for your stack, get in touch — we do this work on Snowflake, Databricks, and Trino platforms every week.