ETL Advisors | Error Handling and Observability for Talend Jobs: tLogCatcher, Joblets, and Structured Logs

Most Talend estates we are asked to review can tell you that something broke. Very few can tell you what broke, on which row, how long it had been degrading, or whether last night's job ran at all. The usual state of play is a tDie here, an emailed stack trace there, and a scheduler log nobody reads until a business user complains that a report is stale.

This tutorial sets out the error-handling and observability pattern we install on engagements. It is deliberately boring: one reusable joblet, one audit table, one structured log format. It works on Talend Studio subscription editions, on Qlik Talend Cloud with Remote Engines, and — with a little more hand-wiring — on legacy Open Studio estates that have not migrated yet.

The three questions a pipeline must answer

Before adding components, be clear about what you are instrumenting for. Every run should answer:

Did it run? Absence of failure is not success. A job that never started produces no error at all.
What did it do? Rows in, rows out, rows rejected, duration, and the context/environment it ran against.
What went wrong, and where? Job name, subjob, component, message, and enough business key to find the offending record.

A tDie with "Error loading customers" answers none of these. Design for the on-call engineer at 3am who has never opened Studio.

1. Catch errors centrally, not per component

Talend gives you three catchers that most teams under-use:

tLogCatcher — receives messages from tWarn and tDie, plus Java exceptions when Catch Java Exception is ticked. Columns include type, origin, priority, message, code.
tStatCatcher — receives job and subjob start/stop events with durations when Statistics is enabled on components or via the job's Stats & Logs tab.
tFlowMeterCatcher — receives row counts from tFlowMeter components you place on key flows.

The pattern is one small subjob per job, not one catcher per component:

tLogCatcher ---> tMap ---> tDBOutput (job_run_log)
                   |
                   +--> tSendMail / webhook  (only when priority >= 5)

In the tMap, enrich every row with what the catcher does not know: pid (Numeric.sequence), the job name (jobName), the project, the environment context variable, and the run identifier described below. Without enrichment your log table is a pile of messages with no way to group a single run.

One trap worth naming: tLogCatcher fires after tDie has already flagged the job as failed, and by default tDie ends the job immediately. If you need the catcher subjob to complete, use tDie with Die on error deselected on the component that fails, or raise the error with tWarn and terminate at the end of the flow.

2. Give every run an identity

Correlation is the whole game. Generate a run ID at the very start of the job and pass it everywhere — into log rows, into audit columns on loaded tables, into the log line prefix.

A custom Java routine is the cleanest place for it:

package routines;

import java.util.UUID;

public class RunContext {

    /** Stable id for one execution of one job tree. */
    public static String newRunId() {
        return UUID.randomUUID().toString();
    }

    /** Child jobs inherit the parent id when one is supplied. */
    public static String runId(String inherited) {
        return (inherited == null || inherited.trim().isEmpty())
                ? newRunId()
                : inherited;
    }
}

Add a run_id context variable to your reference project, set it in a tJava at the top of the job with context.run_id = RunContext.runId(context.run_id);, and pass it down to child jobs through tRunJob context parameters. Combined with context management across environments, you now have a single key that ties a Snowflake load, a reject file, and an alert together.

3. The job-run audit table

Two tables cover almost every reporting need. Keep them narrow so writes never become the bottleneck:

CREATE TABLE etl_job_run (
  run_id        VARCHAR(36)  NOT NULL,
  job_name      VARCHAR(200) NOT NULL,
  parent_run_id VARCHAR(36),
  environment   VARCHAR(20)  NOT NULL,
  started_at    TIMESTAMP    NOT NULL,
  ended_at      TIMESTAMP,
  status        VARCHAR(12)  NOT NULL,   -- RUNNING / SUCCESS / FAILED
  rows_in       BIGINT,
  rows_out      BIGINT,
  rows_rejected BIGINT,
  PRIMARY KEY (run_id, job_name)
);

CREATE TABLE etl_job_event (
  run_id     VARCHAR(36)  NOT NULL,
  event_time TIMESTAMP    NOT NULL,
  severity   VARCHAR(10)  NOT NULL,
  origin     VARCHAR(200),
  code       INT,
  message    VARCHAR(4000)
);

Write a RUNNING row on entry and update it on exit — that is what lets you detect jobs that died so hard they never wrote a failure. Anything still RUNNING well past its normal duration is an incident, and a five-line query finds it:

SELECT job_name, run_id, started_at
FROM   etl_job_run
WHERE  status = 'RUNNING'
AND    started_at < CURRENT_TIMESTAMP - INTERVAL '2' HOUR;

Use a dedicated connection for logging. If the audit write shares the transaction of the load it is auditing, a rollback erases your evidence.

4. Wrap it in a joblet so nobody re-implements it

Copy-pasted logging subjobs drift within a month. Build one joblet — call it jl_JobLogging — containing the catchers, the enrichment tMap, and the audit writes, with joblet context parameters for run_id, job_name and environment. Ship it in a reference project so every project inherits it, and make "uses jl_JobLogging" a code-review checklist item.

Because the joblet lives in the reference project, it is part of your Git sources and flows through the CI/CD pipeline like anything else. Change the log schema once, rebuild, and the whole estate picks it up.

5. Emit structured logs, not prose

Platform teams increasingly ingest engine logs into Splunk, Elastic, Datadog or OpenSearch. Free-text System.out.println is unusable there; single-line JSON is trivially parseable. A small routine keeps job designs clean:

public static String logJson(String runId, String job, String level, String msg) {
    return "{\"ts\":\"" + java.time.Instant.now() + "\""
         + ",\"level\":\"" + level + "\""
         + ",\"job\":\"" + job + "\""
         + ",\"run_id\":\"" + runId + "\""
         + ",\"msg\":\"" + msg.replace("\"", "'") + "\"}";
}

Call it from tJava/tJavaRow or from the tLogCatcher branch via a second output. Never log credentials, tokens or full payload rows — if you are calling OAuth 2.0 APIs from Talend, log the token's expiry, not the token.

6. Reject rows are data, not exceptions

A malformed record in a ten-million-row feed should not fail the job. Route it:

Tick Die on error = off on tMap, database outputs and file inputs, then use the Reject output link.
Land rejects in a table with the same run_id, the raw row, and the reason.
Enforce a threshold: if rejects exceed, say, 1% of input, tDie at the end of the job. Silent partial loads are worse than loud failures.

This distinction — hard failure versus data defect — is what separates a pipeline that people trust from one everybody reruns "just in case".

7. Retry only the things worth retrying

Network blips, API 429s and warehouse queue timeouts are transient. Constraint violations are not. Implement retries narrowly, around the call that fails, with backoff:

tLoop (max 3) --iterate--> tRESTClient --OnComponentOk--> tJava (break)
                                 |
                          OnComponentError
                                 |
                            tSleep (2^n seconds)

Make the operation idempotent before retrying it — a retried insert that has no natural key is how you end up with duplicate facts. Our CDC patterns post covers the merge keys that make this safe.

8. Alert on the runs that never happened

The hardest failure to see is silence: a scheduler that did not fire, a Remote Engine that was down, a paused task. Add a heartbeat check — a small job that runs hourly and asserts that every expected job has a SUCCESS row inside its SLA window:

SELECT s.job_name
FROM   etl_job_schedule s
LEFT   JOIN etl_job_run r
       ON r.job_name = s.job_name
      AND r.status   = 'SUCCESS'
      AND r.ended_at > CURRENT_TIMESTAMP - s.sla_interval
WHERE  r.run_id IS NULL;

Route the result to the same channel your team already watches. Pair it with the scheduling guidance in Scheduling Talend Jobs in 2026 so the SLA table and the schedule cannot drift apart.

A rollout that finishes

Doing this across a large estate is a migration, not a big bang:

Create the tables and the joblet; instrument one medium-importance job end to end.
Add the heartbeat job and the RUNNING-too-long query. You will find broken schedules in week one.
Instrument the top 20 jobs by business impact.
Make the joblet mandatory for new development, and retro-fit the tail during normal maintenance.
Only then build the dashboard — run counts, durations over time, reject rates by source.

Duration trends are also the cheapest performance signal you will ever get: when a job's runtime doubles, you know where to point the tuning work.

Where we can help

We retro-fit logging, audit and alerting frameworks into existing Talend estates — including estates nobody has documented — and we do it without rewriting the jobs themselves. If your integration platform can only tell you that something broke, get in touch and we will scope the instrumentation work with you.