Calling OAuth 2.0 APIs from Talend

Our 2014 tREST tutorial used a marketplace API key copied into a header. That world is gone: nearly every serious API (Salesforce, Microsoft Graph, Google, HubSpot, NetSuite, most internal platforms) now expects an OAuth 2.0 bearer token, and tokens expire. This tutorial shows the complete pattern in current Talend Studio: get a token, use it, page through results, and survive rate limits.

The flow we are implementing

For machine-to-machine integration the right grant is client credentials: your job presents a client ID and secret to the token endpoint and receives a short-lived access token. The sequence is:

1. POST to the token endpoint with grant_type=client_credentials
2. Extract access_token (and expires_in) from the JSON response
3. Call the API with Authorization: Bearer <token>
4. Follow pagination until there are no more pages
5. Back off and retry on 429 / 5xx

Authorization-code flows (the ones with a browser login) are for user-facing apps; for a scheduled Talend job, client credentials or a refresh-token flow is what you want.

Step 1: Contexts

Create a context group api with:

Variable	Type	Notes
token_url	String	e.g. https://login.example.com/oauth2/v2.0/token
client_id	String
client_secret	Password	never in the artifact; load from env or TMC at run time
scope	String	as required by the provider
api_base	String	e.g. https://api.example.com/v1

Use a tContextLoad fed from environment variables in production so the secret is never committed to the project.

Step 2: Fetch the token

Use tRESTClient (the CXF-based client; it handles form bodies and headers cleanly) with:

• URL: context.token_url
• HTTP Method: POST
• Content type: application/x-www-form-urlencoded
• Accept type: JSON
• Body: built in a preceding tFixedFlowInput or tJava as a single string column:

// tJava, before tRESTClient_token
String body = "grant_type=client_credentials"
    + "&client_id=" + java.net.URLEncoder.encode(context.client_id, "UTF-8")
    + "&client_secret=" + java.net.URLEncoder.encode(context.client_secret, "UTF-8")
    + "&scope=" + java.net.URLEncoder.encode(context.scope, "UTF-8");
globalMap.put("token_body", body);

Some providers want client ID and secret as HTTP Basic auth instead of form fields. In that case set the Authorization header to "Basic " + java.util.Base64.getEncoder().encodeToString((context.client_id + ":" + context.client_secret).getBytes("UTF-8")) and send only grant_type and scope in the body. Read the provider's token-endpoint documentation; this is the single most common point of failure.

Step 3: Extract the token with tExtractJSONFields

Connect tRESTClient_token (its Response flow, column string) to tExtractJSONFields with:

• Read by: JsonPath
• Loop JSONPath query: $
• Mapping: access_token -> $.access_token, expires_in -> $.expires_in, token_type -> $.token_type

Then a tJavaRow stores it for the rest of the job:

globalMap.put("access_token", input_row.access_token);
globalMap.put("token_expires_at",
    System.currentTimeMillis() + (Long.parseLong(input_row.expires_in) - 60) * 1000L);

The - 60 seconds gives you a safety margin so a long run does not send a token that expires mid-request.

Step 4: Call the API

A second tRESTClient (or tRESTClient in a loop; see pagination below):

• URL: context.api_base + "/customers"
• HTTP Method: GET
• Accept type: JSON
• HTTP Headers: name Authorization, value "Bearer " + (String) globalMap.get("access_token")
• Query parameters: page_size = "200"

Feed its response into another tExtractJSONFields, this time looping over the records:

• Loop JSONPath query: $.data[*]
• Mapping: id -> id, name -> name, updated_at -> updated_at

and on into your tMap/tDBOutput as usual.

Step 5: Pagination

APIs paginate one of three ways; handle whichever yours uses.

Cursor / next-link. The response carries next (a URL or token). Structure the job as:

tSetGlobalVar (next = first URL)
  --OnSubjobOk--> tLoop (while: globalMap.get("next") != null)
     --Iterate--> tRESTClient_page --Response--> tExtractJSONFields_meta (maps $.next)
                                              --> tJavaRow: globalMap.put("next", input_row.next)
                  tRESTClient_page --Response--> tExtractJSONFields_rows ($.data[*]) --> target

Set tLoop to While with condition globalMap.get("next") != null, and make the page component's URL (String) globalMap.get("next").

Offset / page number. Same tLoop, but as a For loop, or a While loop that stops when a page returns fewer than page_size rows (count them with tAggregateRow or increment a counter in tJavaRow).

Link header (RFC 5988). Parse the Link response header in a tJavaRow on the response flow with a regex for <(.*?)>; rel="next".

Step 6: Token expiry on long runs

Before each page request, check the stored expiry and refresh if needed. A small child job get_token that performs Steps 2 and 3 and is called via tRunJob keeps this clean:

// tJava at the top of each loop iteration
Long exp = (Long) globalMap.get("token_expires_at");
if (exp == null || System.currentTimeMillis() > exp) {
    globalMap.put("refresh_token_needed", true);
} else {
    globalMap.put("refresh_token_needed", false);
}

then an If trigger ((Boolean) globalMap.get("refresh_token_needed")) into the tRunJob for get_token. Pass the token back through a context variable marked as an output, or have the child write to a shared tHashOutput.

Step 7: Rate limits and retries

Enable Die on error off on the API tRESTClient and route its Reject/error flow to logic that inspects the status code. Handle:

• 429 Too Many Requests: read Retry-After (seconds) from the headers if present, otherwise back off exponentially (1s, 2s, 4s, up to a cap), then retry the same page.
• 5xx: same backoff, with a retry limit (three to five attempts) before failing the job.
• 401: refresh the token once and retry; a second 401 is a configuration error, fail loudly.

A simple implementation uses a tLoop around the request with a tSleep whose duration comes from a globalMap counter the tJavaRow doubles on each failure. Keep the total retry budget bounded so a dead API does not hold your schedule hostage.

Worked example with a public API

To test the shape of the job without a vendor account, Open-Meteo is keyless (no token step) and JSON-based, so Steps 4 and 5 can be exercised directly:

https://api.open-meteo.com/v1/forecast?latitude=29.42&longitude=-98.49&current=temperature_2m,wind_speed_10m

Point tRESTClient at that URL with no Authorization header, set the loop query in tExtractJSONFields to $.current, map temperature_2m and wind_speed_10m, and run. Once the extraction path works, add the token subjob and the Authorization header for your real provider. Our refreshed tREST tutorial walks through the keyless version step by step.

Checklist

• Secrets loaded at run time, never stored in the project
• Token refreshed before expiry, with a margin
• Pagination loop terminates on the provider's signal, not on a guessed count
• 429/5xx handled with bounded backoff; 401 triggers exactly one refresh
• Response bodies logged only at debug level (they contain tokens and data)

Component reference: Talend REST components.

Need an API integration built or an old one brought up to OAuth 2.0? Contact us.