Data Lineage for Talend Jobs: OpenLineage Events, Marquez, and Column-Level Detail

Every data platform review we walk into in 2026 asks the same question: where did this column come from? dbt answers it. Airflow answers it. Snowflake and Databricks answer it for what happens inside them. The Talend jobs sitting between the source systems and the warehouse are usually the one hop the lineage graph cannot see — which is exactly the hop the auditor cares about.

The good news is that you do not need a new tool. OpenLineage is an open standard for emitting lineage events over HTTP, and Talend can speak it with nothing more exotic than a tRESTClient and a joblet. This tutorial shows the full pattern: run-level events, dataset facets, column-level mappings pulled out of your tMap logic, and where the result shows up in Marquez or DataHub.

What OpenLineage actually expects

An OpenLineage producer sends JSON RunEvent documents to POST /api/v1/lineage. Each event carries:

Field	Meaning for a Talend job
eventType	START, COMPLETE, FAIL, or ABORT
eventTime	ISO-8601 timestamp, UTC
run.runId	A UUID — use one per job execution
job.namespace / job.name	e.g. talend://prod-tac and CustomerLoad_Daily
inputs[] / outputs[]	Datasets, each with its own namespace and name
producer	A URI identifying your emitter

The contract is deliberately small. A START event at the top of a job and a COMPLETE (or FAIL) event at the end is a valid, useful lineage feed. Everything else — schemas, column lineage, row counts — is a facet layered on top.

Step 1: A lineage joblet

Do not copy REST components into 400 jobs. Build one joblet, jl_OpenLineage, driven by context variables:

Variable	Type	Example
ol_url	String	http://marquez.internal:5000/api/v1/lineage
ol_namespace	String	talend://prod
ol_api_key	Password	bearer token if your collector needs one
ol_enabled	Boolean	lets you switch emission off per environment

Inside the joblet:

1. tJava builds the JSON payload into a global variable.
2. tFixedFlowInput turns that string into a single row.
3. tRESTClient POSTs it with Content-Type: application/json.
4. tLogRow / tWarn on non-2xx — never let lineage emission fail the job.

That last point matters more than anything else in this article. Lineage is observability, not payload. Wrap the REST call so a collector outage degrades to a warning:

// tJava after the tRESTClient, reading its status code
if (((Integer)globalMap.get("tRESTClient_1_ERROR_CODE")) != null) {
    System.err.println("OpenLineage emit failed - continuing");
}

Step 2: Emitting the START event

Generate the run ID once, in the parent job, and pass it down so every subjob shares it:

// tJava at the start of the parent job
String runId = java.util.UUID.randomUUID().toString();
globalMap.put("OL_RUN_ID", runId);
globalMap.put("OL_START", java.time.Instant.now().toString());

Then build the event body. Keep it in a Java routine (OpenLineageUtil) rather than a giant string inside tJava — see our Java routines tutorial for how to add one to a project:

public static String startEvent(String ns, String job, String runId,
                                String inputs, String outputs) {
    return "{"
      + "\"eventType\":\"START\","
      + "\"eventTime\":\"" + java.time.Instant.now().toString() + "\","
      + "\"producer\":\"https://etladvisors.com/talend-openlineage/1.0\","
      + "\"run\":{\"runId\":\"" + runId + "\"},"
      + "\"job\":{\"namespace\":\"" + ns + "\",\"name\":\"" + job + "\"},"
      + "\"inputs\":[" + inputs + "],"
      + "\"outputs\":[" + outputs + "]"
      + "}";
}

Use jobName (the built-in Talend variable) for job.name so a renamed job does not silently fork its lineage history.

Step 3: Naming datasets so the graph actually joins up

This is where most first attempts fail. A lineage graph is only useful if the dataset your Talend job writes has the same identifier as the dataset dbt or Snowflake reads. OpenLineage has naming conventions; follow them exactly:

Source	Namespace	Name
Snowflake	snowflake://myacct.eu-west-1	ANALYTICS.RAW.CUSTOMER
PostgreSQL	postgres://db-host:5432	warehouse.public.orders
S3	s3://landing-bucket	inbound/crm/customers/
SFTP file	file://sftp-host	/out/daily/orders.csv
Kafka	kafka://broker:9092	crm.customer.v1

Store these strings as context variables next to the connection contexts they describe. If you already manage multiple database environments with contexts, add an ol_dataset variable to the same group — the lineage identity then travels with the connection, and dev events land in the dev namespace automatically.

Step 4: Column-level lineage from tMap

Run-level lineage answers "which job touched this table". Column-level answers "which source field produced CUSTOMER.RISK_BAND", which is the question compliance teams ask. OpenLineage expresses it with the columnLineage output facet:

"outputs": [{
  "namespace": "snowflake://myacct.eu-west-1",
  "name": "ANALYTICS.RAW.CUSTOMER",
  "facets": {
    "columnLineage": {
      "_producer": "https://etladvisors.com/talend-openlineage/1.0",
      "_schemaURL": "https://openlineage.io/spec/facets/1-0-1/ColumnLineageDatasetFacet.json",
      "fields": {
        "RISK_BAND": {
          "inputFields": [
            {"namespace":"postgres://db:5432","name":"crm.public.account","field":"credit_score"},
            {"namespace":"postgres://db:5432","name":"crm.public.account","field":"tenure_months"}
          ],
          "transformationType": "INDIRECT",
          "transformationDescription": "tMap expression: risk banding"
        }
      }
    }
  }
}

You do not hand-write that. The mappings already exist inside the job's .item file, which is XML. A small parser over your project directory can walk every tMap node, read each output column's expression, and extract the row1.credit_score-style references:

// sketch: run as a build-time job, not at runtime
// 1. tFileList over <project>/process/**/*.item
// 2. tFileInputXML, loop on //node[@componentName='tMap']
// 3. read elementParameter name='MAP_EXPRESSIONS' or the mapper data model
// 4. regex out ([A-Za-z0-9_]+)\.([A-Za-z0-9_]+) per output expression
// 5. write a JSON sidecar keyed by job name

The job then loads its own sidecar at runtime and attaches it to the COMPLETE event. Direct passthroughs (row1.email → EMAIL) map to transformationType: IDENTITY; anything with a function, lookup or condition in it is INDIRECT. Parsing .item XML is the same technique we use when auditing a thousand-job estate before a migration, so if you already have that inventory script, you are most of the way there.

Step 5: COMPLETE, FAIL, and row counts

On the success path, emit COMPLETE with an outputStatistics facet so the graph carries volume as well as shape:

"facets": {
  "outputStatistics": {
    "rowCount": 184233,
    "size": 0,
    "_producer": "https://etladvisors.com/talend-openlineage/1.0"
  }
}

Pull rowCount from ((Integer)globalMap.get("tSnowflakeOutput_1_NB_LINE")) or the equivalent NB_LINE_INSERTED variable for your writer.

On the failure path, hang the joblet off your existing tLogCatcher subjob and send eventType: FAIL with an errorMessage run facet. If you followed our error handling and observability patterns, the catcher is already central and this is a two-component change. A lineage graph that only records successes will quietly lie to you: stale downstream tables look freshly loaded.

Step 6: Parent/child runs across job calls

A tRunJob chain should show up as a nested run, not as three unrelated jobs. Pass the parent run ID into the child through a context variable and emit a parent run facet:

"run": {
  "runId": "9f1c...child",
  "facets": {
    "parent": {
      "run": {"runId": "3ab2...parent"},
      "job": {"namespace": "talend://prod", "name": "MasterLoad_Daily"}
    }
  }
}

Marquez renders that as a collapsible run tree. If your orchestration lives outside Talend — Airflow, Control-M, a Kubernetes CronJob — set the parent facet from the scheduler's run ID instead, and the Talend hop stitches directly into the orchestration graph. That also plays well with the patterns in scheduling Talend jobs in 2026 and containerizing Talend jobs, where the run ID is already an environment variable.

Step 7: Where the events land

Marquez is the reference implementation and the fastest way to prove the pattern: run the container, point ol_url at it, execute one job, and you have a graph in minutes. It is excellent for engineering, thinner for governance.

DataHub and OpenMetadata both ingest OpenLineage events natively, so the same emitter feeds a real catalog with ownership, glossary terms and PII tags. Microsoft Purview and Unity Catalog can consume lineage too, though usually through their own connectors — emit OpenLineage first, bridge second, so you are never locked to one catalog.

Whatever the sink, put a collector in front of it (a small HTTP endpoint or an event topic) rather than letting 400 production jobs POST straight to your catalog. Collectors buffer; catalogs have maintenance windows.

What we recommend doing first

Do not attempt column-level lineage across the whole estate in week one. The order that works:

1. Joblet + run-level START/COMPLETE/FAIL on your ten most business-critical jobs.
2. Correct dataset naming, verified by checking that the Talend node and the warehouse node merge in the graph rather than sitting side by side.
3. Row-count facets, so freshness and volume alerts become possible.
4. Column lineage from .item parsing, for the jobs that touch regulated columns only.

Step 2 is the one people skip and the one that determines whether anybody trusts the graph. Ninety percent of the value arrives by step 3.

---

If your Talend estate is the unlit section of your lineage graph — or an audit is asking where a regulated field comes from and the honest answer is "a tMap somewhere" — get in touch. Our consultants do this instrumentation as a fixed-scope engagement, joblet and parser included.