Teams leaving Talend Administration Center and on-prem JobServers behind usually assume the cloud control plane takes over everything. It does not. In Qlik Talend Cloud the control plane schedules, versions, and monitors work, but the actual data movement still happens on a Remote Engine that you install, size, patch, and pay for. Every hybrid migration we have run in the last two years has hit the same handful of surprises at the engine layer.
This is a practical walkthrough of how to plan that layer before you cut over.
What the Remote Engine actually is
A Remote Engine is a runtime process you install on a host you control — a VM in your data centre, an EC2 instance, or a container in your own cluster. It:
- polls the cloud control plane over outbound HTTPS for work,
- pulls the artifact (a built job or pipeline) into a local workspace,
- executes it against your databases, files, and APIs,
- streams execution logs and status back up to the control plane.
The important property: your data never has to traverse the vendor's plane to get from source to target. Rows read from an on-prem Oracle instance and written to Snowflake go straight from your engine host to Snowflake. That is the whole argument for the hybrid pattern, and it is also why the engine host becomes a piece of production infrastructure with real capacity requirements.
Sizing: start from concurrency, not from row counts
The question people ask is "how big a box do we need for 40 million rows?" The better question is "how many jobs run at the same time, and what is the heaviest one?"
Each concurrent task on a Remote Engine is a JVM. Budget roughly:
| Workload profile | Heap per task | Notes |
|---|---|---|
| Streaming row-by-row (DB to DB, bulk loaders) | 512 MB – 1 GB | Memory flat regardless of volume |
| tMap with a lookup loaded in memory | 1 – 4 GB | Sized by lookup width x rows, not by main flow |
| tSortRow / tAggregateRow / tUniqRow on a large flow | 2 – 8 GB | Use temp-disk options to cap this |
| XML/JSON DOM parsing of large documents | 2 – 6 GB | Prefer streaming/XPath loops instead |
Then: host RAM = (max concurrent tasks x average heap) + 4 GB for the engine and OS. Do not size for the average; size for the nightly window where twelve jobs fire at 02:00.
CPU matters less than people expect. Most ETL tasks are I/O-bound and a single job pins about one core. Two to four vCPU per concurrent task slot is generous. Disk is the sleeper: artifact workspaces, temp sort files, and staged extract files all land on local disk. We give engine hosts a dedicated 100 GB+ volume for the working directory and monitor it, because a full disk fails jobs in a way that looks like random component errors.
A reasonable starting point for a mid-size shop: 8 vCPU, 32 GB RAM, 200 GB SSD, sized for six concurrent tasks, and set the engine's parallel-execution limit explicitly rather than leaving it at the default. An engine that accepts more work than it has memory for will OOM-kill jobs that were fine yesterday.
Networking: outbound only, and prove it
The engine opens outbound HTTPS (443) to the cloud region's endpoints and keeps that channel alive. There is no inbound firewall rule, no DMZ host, no NAT hairpin. Security reviews go smoother when you show them that.
What to nail down before install:
- Egress allow-list. If outbound traffic goes through a proxy or a restrictive egress firewall, get the region's endpoint hostnames allow-listed first. Engines that cannot reach the control plane look "installed" but never accept a task.
- Proxy configuration. Corporate proxies need to be configured in the engine's configuration file, and the proxy must not MITM the TLS session unless you have imported its CA into the engine JVM truststore. Half the failed installs we see are a proxy CA problem misread as an authentication problem.
- Source and target reachability. The engine host needs network paths to every database, file share, SFTP endpoint, and API it will touch. Draw that matrix before you pick where the host lives. An engine in the wrong VPC subnet is the single most common cause of a stalled cutover.
- Placement. Put the engine close to the heaviest data source. Pulling 200 GB across a VPN to an engine that then pushes it to a cloud warehouse in the same region as the source is a self-inflicted wound.
Secrets: stop putting credentials in contexts
On-prem, most shops shipped .properties files or implicit context loads from a config table. In a hybrid setup you have three legitimate places for credentials, and you should pick one deliberately:
- Cloud-managed connections. Defined once in the control plane, referenced by jobs, resolved at runtime on the engine. Good for shared targets like the warehouse.
- Engine-local environment or vault. For secrets that must never leave your network, keep them on the host — an env var injected from HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault at engine start — and have jobs read them via a context variable populated from
System.getenv(). - Per-run parameters. For rotating or per-tenant secrets, pass them at task invocation.
What you should not do is keep shipping credentials inside the built artifact. Artifacts are stored, versioned, and downloadable; treat them as if they will be read.
Engine pools and workload isolation
One engine for everything is fine until the day a memory-hungry reconciliation job starves your hourly incremental loads. Two patterns are worth the extra hosts:
- Split by criticality. A small engine dedicated to short, frequent, business-critical tasks; a larger one for the heavy batch window. Nothing that runs for two hours can block a five-minute task.
- Split by network zone. One engine inside the restricted database zone, one in the general application zone. Each only gets the routes it genuinely needs, which is an easier conversation with the network team than one host with routes to everything.
Where the platform supports engine clustering, put two hosts behind the same logical target so a patched or rebooted host does not mean a missed schedule. Even without clustering, a warm standby engine with the same configuration turns an outage into a five-minute failover.
Upgrades and drift
Remote Engines are versioned software with a supported-version window, and the control plane will eventually stop accepting old ones. Two habits keep this boring:
- Treat the host as cattle. Build the engine host from a script or image — install, configure, register with a pre-provisioned token, start. Then an upgrade is a rebuild, not a surgery. This also solves the "nobody remembers what got installed on that box in 2021" problem.
- Pin and test the JDK. The engine and your jobs run on a specific Java runtime. Custom routines, third-party JARs added via tLibraryLoad, and JDBC drivers all care. Upgrade the JDK in a non-production engine first and run a representative job set before touching production.
Keep an inventory of external JARs your jobs need on the engine host. Those do not travel with the cloud control plane, and a job that runs fine on the developer's Studio will fail on a fresh engine that is missing an Oracle or SAP driver.
Monitoring: watch the engine, not just the jobs
The control plane will tell you a task failed. It will not tell you the host is at 95% disk or that the engine process has been restarting in a loop. Add to your normal monitoring:
- engine process up/down and connection state to the control plane,
- host CPU, memory, and working-directory disk usage,
- count of queued vs. running tasks (a growing queue means you are under-provisioned),
- JVM GC pressure on the heaviest recurring tasks.
Pair that with the structured logging patterns you already use inside jobs — a run ID, row counts in and out, and a duration written to a run-log table — so you can tell "the job was slow" apart from "the engine was saturated."
A sane cutover sequence
- Stand up one Remote Engine in a non-production zone, register it, and run a trivial job end-to-end. Prove outbound connectivity and proxy/TLS before anything else.
- Port a single medium-complexity job. Fix the driver, secret, and network gaps it exposes — there will be some.
- Run production and the new engine in parallel for a full business cycle, comparing row counts and durations. Do not decommission the old JobServer on the strength of one green run.
- Migrate by schedule group, heaviest window last, so the sizing assumptions get tested by the batch that actually matters.
- Only then turn off the old scheduler — and keep the host around, powered off, for a month.
The short version
Hybrid does not mean less operations work; it means different operations work. You trade a scheduler you owned for a control plane you do not, and in exchange you keep full control of where your data actually flows. Size the engines for peak concurrency rather than average volume, keep egress and secrets deliberate, split workloads across engines before they collide, and monitor the host as seriously as you monitor the jobs.
If you are planning a move from Talend Administration Center to a cloud control plane and want a second opinion on engine topology or the cutover sequence, get in touch — it is a conversation we have had many times.