+1 (726) 227-3027
ETL Advisors

Automating Qlik Talend Cloud: Promotions, Runs, and Monitoring with the Public API

Most teams we meet publish Talend artifacts from Studio, then click through Talend Management Console (TMC) to promote them from DEV to TEST to PROD, attach a run profile, and hit Go. That works fine at ten jobs. At two hundred jobs across three environments it is a full-time job and an audit problem: nobody can tell you six months later who promoted what, or why Tuesday's load ran twice.

Everything the TMC screen does is available on the Qlik Talend Cloud public API. This post is the automation layer we put on top of it: token handling, idempotent promotion, task execution, and run monitoring that feeds your existing alerting.

Before you start

You need three things:

  1. A service account, not a personal token. Create it in TMC under Users > Service accounts and give it the Integration Operator and Integration Developer roles for the environments it will touch. Personal tokens die when the person leaves; we have cleaned up that outage more than once.
  2. The right regional base URL. The API is regional. https://api.<region>.cloud.talend.com — for example api.eu.cloud.talend.com or api.us.cloud.talend.com. Calling the wrong region returns a confusing 401, not a 404.
  3. A place to keep the secret. Vault, AWS Secrets Manager, GitHub Actions secrets — anywhere but a context file in Git.

Step 1: get a token

Service accounts exchange an id/secret pair for a short-lived access token:

TOKEN=$(curl -s -X POST "https://api.eu.cloud.talend.com/security/oauth/token" \
  -H "Content-Type: application/json" \
  -d "{\"grant_type\":\"client_credentials\",\"client_id\":\"$TMC_CLIENT_ID\",\"client_secret\":\"$TMC_CLIENT_SECRET\",\"audience\":\"https://api.eu.cloud.talend.com\"}" \
  | jq -r .access_token)

Tokens expire (typically within the hour). Fetch one per pipeline run and re-fetch on any 401; do not cache it across a long-running orchestrator process without a refresh path.

Every call below sends Authorization: Bearer $TOKEN.

Step 2: find your environment and workspace IDs

Nothing in the API takes names. Resolve them once and store the IDs as pipeline variables:

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://api.eu.cloud.talend.com/orchestration/environments" \
  | jq -r '.items[] | "\(.name)\t\(.id)"'

curl -s -H "Authorization: Bearer $TOKEN" \
  "https://api.eu.cloud.talend.com/orchestration/workspaces?environmentId=$ENV_ID" \
  | jq -r '.items[] | "\(.name)\t\(.id)"'

A useful convention: one workspace per delivery stream (Finance, CRM, Regulatory), identically named in every environment. Promotion scripts then map DEV workspace Finance to PROD workspace Finance by name and you never hand-maintain an ID table.

Step 3: promote an artifact between environments

Promotion is asynchronous. You submit a request and then poll it:

EXEC=$(curl -s -X POST "https://api.eu.cloud.talend.com/processing/executables/promotions" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{
        "sourceEnvironmentId": "'"$DEV_ENV"'",
        "targetEnvironmentId": "'"$PRD_ENV"'",
        "artifacts": [{"id": "'"$ARTIFACT_ID"'", "version": "'"$VERSION"'"}],
        "overwriteExisting": true
      }' | jq -r .executionId)

while :; do
  ST=$(curl -s -H "Authorization: Bearer $TOKEN" \
        "https://api.eu.cloud.talend.com/processing/executions/$EXEC" | jq -r .status)
  case "$ST" in
    SUCCESSFUL) echo "promoted"; break ;;
    FAILED|KILLED) echo "promotion $ST"; exit 1 ;;
    *) sleep 5 ;;
  esac
done

Two details that bite people:

  • Promote a pinned version, never "latest". Your CI job already knows the Maven version it published; pass it explicitly. "Latest" means a concurrent developer publish can ride into PROD with your release.
  • Promotion moves the artifact, not the configuration. Connections, resources, and context values live per environment. That is the point — but it means a new context parameter added in DEV must be created in the PROD environment before the promotion, or the first run fails with a null parameter. We add a pre-flight check that diffs the artifact's declared parameters against the target environment's stored values and fails the pipeline early.

Step 4: create or update the task

An artifact does nothing until a task binds it to a workspace, a runtime (remote engine or cloud engine), and a parameter set. Make this idempotent: look for a task with your naming convention, create it if missing, update it if present.

TASK_ID=$(curl -s -H "Authorization: Bearer $TOKEN" \
  "https://api.eu.cloud.talend.com/orchestration/executables/tasks?workspaceId=$WS_ID" \
  | jq -r --arg n "$TASK_NAME" '.items[] | select(.name==$n) | .executable')

if [ -z "$TASK_ID" ]; then
  TASK_ID=$(curl -s -X POST "https://api.eu.cloud.talend.com/orchestration/executables/tasks" \
    -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    -d '{"name":"'"$TASK_NAME"'","workspaceId":"'"$WS_ID"'","artifact":{"id":"'"$ARTIFACT_ID"'","version":"'"$VERSION"'"},"runtime":{"type":"REMOTE_ENGINE","id":"'"$ENGINE_ID"'"}}' \
    | jq -r .executable)
else
  curl -s -X PUT "https://api.eu.cloud.talend.com/orchestration/executables/tasks/$TASK_ID" \
    -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    -d '{"version":"'"$VERSION"'"}' > /dev/null
fi

Keep the task name deterministic — {stream}_{jobname}_{schedule} — because the name is the only stable handle a human will recognise in the console at 3 a.m.

Step 5: run it and wait honestly

RUN=$(curl -s -X POST "https://api.eu.cloud.talend.com/processing/executions" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"executable":"'"$TASK_ID"'","parameters":{"loadDate":"'"$LOAD_DATE"'"}}' \
  | jq -r .executionId)

Then poll /processing/executions/{id} until the status leaves RUNNING. Three rules for the polling loop:

  • Back off. Poll every 5 seconds for the first minute, then every 30. Hammering the API gets you rate-limited (429), and a rate-limited monitor looks exactly like a failed job.
  • Set a ceiling. A job that has run for three times its p95 duration is an incident, not a long run. Break the loop, raise an alert, and leave the execution alone — killing it from a script usually creates a half-loaded target table.
  • Treat 429, 502, and 504 as retryable; treat 400 and 403 as bugs in your script. Retrying a 403 forever is the most common way these automations wedge silently.

Step 6: monitor without building a second console

You do not need a dashboard. You need the failures to reach the on-call channel with enough context to act. A small scheduled poller is enough:

SINCE=$(date -u -d '15 minutes ago' +%Y-%m-%dT%H:%M:%SZ)
curl -s -H "Authorization: Bearer $TOKEN" \
  "https://api.eu.cloud.talend.com/processing/executions?environmentId=$PRD_ENV&from=$SINCE" \
  | jq -r '.items[] | select(.status=="FAILED") |
           "FAILED \(.executable.name) exec=\(.executionId) started=\(.startTimestamp)"'

Pipe that into Slack or PagerDuty with a deep link of the form https://tmc.<region>.cloud.talend.com/#/executions/<executionId>. Pull the run log via the execution's log endpoint and attach the last 40 lines — the person who gets paged should not have to log in to find out whether it was a source outage or a genuine data error.

If your jobs already emit structured logs from tLogCatcher (see our post on error handling and observability), join on the execution ID and you get engine-level status and row-level failure detail in one alert.

What to automate first

If you are starting from a fully manual process, the order that pays back fastest is:

  1. Monitoring. Read-only, zero risk, immediate value.
  2. Task creation/update. Kills the "someone pointed PROD at the DEV artifact" class of incident.
  3. Promotion. Only once your pre-flight parameter check exists.
  4. Scheduling. Last, and only if your orchestrator (Airflow, Control-M, Dagster) is the system of record for dependencies — otherwise leave TMC triggers alone.

The whole thing is about 200 lines of shell or a small Python module. The payoff is not speed; it is that every PROD change has a commit, a reviewer, and a log line.

Need this built properly? ETL Advisors has run Talend and Qlik Talend Cloud delivery pipelines for regulated environments where every promotion needs an audit trail. Get in touch if you want a hand wiring yours up.