ETL Advisors | High Availability and Disaster Recovery for Talend: Remote Engines, JobServers, and a Failover Runbook That Works

Every Talend estate has a runbook for job failures. Far fewer have one for the day the box that runs the jobs is gone. After the TAC and Open Studio end-of-life churn, a lot of shops are running a hybrid estate — Talend Management Console (TMC) in the cloud, Remote Engines or JobServers on their own infrastructure — and nobody has written down what happens when the engine host dies at 2 a.m. on a quarter close.

This is the runbook we build on engagements: what to protect, how to fail over, and how to prove the RTO is real instead of aspirational.

Step 1: Decide the RTO and RPO per pipeline, not per platform

"The platform must be highly available" is not a requirement; it is a budget request. Rate each pipeline family instead:

Pipeline family	Business impact of a 4-hour outage	Target RTO	Target RPO
Regulatory / close-of-business feeds	Reportable breach, fines	1 hour	0 (no lost records)
Customer-facing API routes / data services	Visible outage	15 min	0
Warehouse batch loads	Stale dashboards	8 hours	last successful run
Internal enrichment / backfills	Nobody notices today	72 hours	last backup

Most estates find that 10–15% of jobs justify active-active runtime, and the rest are happy with a rebuild-and-rerun story. That split is the whole design.

Step 2: Inventory what is actually stateful

The parts of a Talend estate that cannot be recreated from source control are short, and they are the parts people forget:

Git repository of projects/jobs — recreatable only if it is mirrored off the primary host.
Artifact repository (Nexus/Artifactory, or TMC's artifact store) — the built JARs that are actually running in production.
Context/parameter values and secrets — vault paths, TMC environment variables, .properties files baked into deploy directories.
Scheduler state — TMC triggers, cron entries, Airflow DAG definitions, plan dependencies.
Run-state tables — high-water marks, CDC offsets, tContextDump audit tables, checkpoint files. This is the real RPO driver.
In-flight file landing zones — SFTP pickups that have been moved but not yet processed.
Engine registration tokens — a Remote Engine's pairing key; you cannot clone a VM and expect a clean second engine.

Everything else — Studio installs, JVMs, OS packages — should be built from an image or an Ansible/Terraform definition, not restored from a backup.

Step 3: Make the runtime tier replaceable

Remote Engines (Qlik Talend Cloud)

Run at least two Remote Engines in the same run profile rather than one big one. TMC distributes executions across available engines in a profile, so a lost engine degrades throughput instead of stopping work — provided:

Both engines can reach every source and target (same firewall rules, same NAT egress IP allowlisted, same private link endpoints). An engine that cannot reach Snowflake is not a failover target.
Each engine is separately paired. Build engine #2 from the same automation but register it with its own pairing token; never snapshot-and-clone a paired engine.
Engine config (org.talend.remote.jobserver.cfg, JVM heap, proxy settings) lives in configuration management, not in someone's ~/notes.txt.
Local filesystem dependencies are eliminated. Any job that reads /data/incoming on engine 1 will silently find nothing on engine 2. Move shared paths to NFS/EFS or object storage.

Spread the pair across two availability zones. Across two regions only if your RTO is under an hour and your sources are multi-region too.

JobServers / on-prem Talend Runtime

For estates still on JobServers or Talend Runtime (Karaf) containers:

Put two JobServers behind the same execution plan and use the scheduler, not a load balancer, to choose. Talend batch execution is not session-based, so an HTTP LB in front adds nothing useful.
For ESB routes that do accept traffic, an LB is correct — health-check the Camel route endpoint (/health on a dedicated route), not just the TCP port, so a Karaf container with failed bundles is taken out of rotation.
Keep the deploy directory identical and deployed by CI, so "fail over" means "point the trigger at host B," not "copy some JARs."

Step 4: Write jobs that survive being killed mid-flight

High availability at the infrastructure layer is wasted if reruns corrupt data. The non-negotiables:

Idempotent writes. MERGE on a business key, or delete-by-partition then insert. Never bare INSERT into a target that a rerun will touch again.
Commit the watermark with the data, in the same transaction where the platform allows it. If the high-water mark is updated in a separate job step, a mid-run kill produces either silent data loss or a duplicate window.
Stage then swap. Load to target_stg, validate counts, then atomically swap or merge. A killed job leaves a junk staging table, not a half-loaded fact table.
Move files, do not delete them. SFTP/landing pickups should move to processing/ with a run ID, then to archive/ on success. On restart, the processing/ directory tells you exactly what was in flight.
Kafka consumers: commit offsets after the sink flush, not before. At-least-once plus an idempotent sink is the only honest guarantee.
Externalise the run ID. Pass it in as a context variable so a retry with the same run ID can clean up its own partial output.

A job that satisfies these can be rerun blindly by the on-call engineer, which is what actually shortens your RTO.

Step 5: Protect the control and artifact planes

Mirror Git to a second provider or a second region. A pull-mirror job every 15 minutes is enough for most shops.
Back up the artifact repository, or be able to rebuild every artifact from a tag. Verify by rebuilding one quarterly — Maven builds rot when a plugin version disappears.
Export TMC configuration (engines, environments, plans, triggers, connections) through the public Management API on a schedule and commit the JSON. This gives you both a DR artifact and a diff when someone changes a trigger by hand.
For self-managed TAC that still exists: back up the TAC database and the config directory together. They must be restored as a matched pair.
Keep secrets in a vault with its own replication story. "The Talend engine is up but nothing can authenticate" is the most common half-failed failover.

Step 6: A failover runbook worth paging someone with

Keep it to one page per scenario. For "primary Remote Engine lost":

Confirm the engine is actually down in TMC (Engines view) and not just network-partitioned from the console.
Pause the affected run profile's triggers so retries do not pile up.
Confirm engine #2 is Running and has the current artifacts and context values.
Re-point the run profile / plan to engine #2 (or confirm auto-distribution picked it up).
Identify in-flight runs at failure time: query the execution log via the Management API, and check processing/ landing directories.
For each in-flight run, apply the rerun rule for that pipeline family (rerun-from-scratch, rerun-from-watermark, or skip and backfill).
Resume triggers.
Reconcile: row counts versus source, watermark continuity, duplicate-key check on the top-5 targets.
Only then start rebuilding the lost engine.

Steps 5, 6 and 8 are where teams lose hours. Write the reconciliation queries now, store them in the repo next to the runbook, and have them take a time window as a parameter.

Step 7: Test it on a schedule

A DR plan that has never been executed is a document, not a capability. Minimum viable testing:

Quarterly: kill the primary engine in non-production during a batch window and run the full runbook with a stopwatch. Record actual RTO.
Quarterly: restore the artifact repo or rebuild all artifacts from tags into a clean Nexus.
Semi-annually: restore TMC configuration from exported JSON into a sandbox environment.
Per release: run one rerun-safety test — kill a representative job at 60% completion, rerun, assert zero duplicates and correct counts. This belongs in CI, not in a calendar reminder.

Publish measured RTO against target RTO per pipeline family. When the gap is visible, funding for the second engine tends to appear.

Common anti-patterns

Cloning a paired Remote Engine VM. Two engines with the same identity fight in TMC; executions go missing. Always pair fresh.
One engine, big instance, "it's on a cluster." VM-level HA restarts the host in minutes but does nothing for in-flight jobs or for a corrupted deploy directory.
Failover that needs a firewall change. If engine #2 needs a ticket to reach production Oracle, your RTO is the ticket queue's SLA.
Backups of the Talend install directory instead of the repository. You are backing up the easy part.
Scheduling state only in TMC's UI. Undocumented triggers are the single most common thing missed after a restore.

Where this lands on the maturity curve

If you can answer "which jobs were running when the engine died, and can we rerun them safely?" in under ten minutes, you have a real DR posture. If the answer involves reading a log on a host that no longer exists, you have a document.

We build this runbook as part of Qlik Talend Cloud and hybrid-runtime engagements — engine topology, rerun-safety review of the job estate, Management API configuration export, and a tested failover drill. If you are carrying a single-engine estate with regulatory feeds on it, that is the gap worth closing first.