Extracting SAP Data with Talend: OData, RFC/BAPI, IDocs, and the Licence Conversation

SAP shows up on most enterprise integration engagements eventually, and it is the extract that derails schedules. Not because the data is exotic, but because the access path is a negotiation between your data team, a Basis team, and an SAP licence agreement. This is the practical version: the four ways to get SAP data into Talend, what each one costs you in approvals and runtime, and the patterns that keep nightly extracts from falling over after an S/4HANA upgrade.

First, settle the access path before you open Studio

Four realistic options, in rough order of how often we end up using them:

1. OData / SAP Gateway services — HTTP, token-based, easy to reach from tRESTClient. Best for moderate volumes and for S/4HANA, where exposing a CDS view as OData is a routine request.
2. CDS views read through a supported SQL path — fast and semantically correct (currency conversion, associations, authorisation already applied), but requires an approved connection and, in most contracts, an SAP licence discussion.
3. RFC / BAPI calls via Talend's SAP components (tSAPConnection, tSAPInput, tSAPBapi) — the classic route for ECC, still very much alive.
4. IDocs dropped to a file or queue — the right answer for event-driven, document-shaped data (orders, deliveries, invoices) that SAP already knows how to push.

Pick before you build. Rewriting an extract because Basis refuses direct table reads is a two-week tax, and it is avoidable with one meeting.

The licensing conversation you cannot skip

Talend's SAP connectors need the SAP Java Connector (JCo) libraries — sapjco3.jar plus the native library for your JobServer's OS. Those are downloaded from SAP under your own agreement; they are not redistributed with Talend, and they cannot be committed into your Git repo. Two consequences:

• Your build needs JCo installed into your Maven repository as a provided dependency, and your runtime image needs the native .so/.dll on java.library.path. Do this once in the Docker image or JobServer bootstrap, not per job.
• Direct reads of SAP application tables through generic function modules (the RFC_READ_TABLE family) are a known grey area in many agreements, and the function module itself has hard limits — row width truncation around 512 bytes, clumsy WHERE clause handling, no support for some data types. It works for small lookups. It is not an extraction strategy for a 40-million-row table.

Say this out loud to the client's architect early. It changes the estimate honestly rather than late.

Pattern 1: OData extraction with paging that resumes

For S/4HANA, ask for a CDS view exposed as an OData v4 service with a delta token enabled. Then the Talend job is unremarkable, which is the point:

• tSetGlobalVar / context holds the service root, entity set, and the last stored delta token.
• tRESTClient issues GET /sap/opu/odata4/.../EntitySet?$select=...&$top=5000 with the token on subsequent runs.
• tExtractJSONFields flattens the payload; @odata.nextLink drives a tLoop or a tFlowToIterate until it is absent.
• Persist the new delta token only after the target load commits, in the same transaction scope if your target supports it. Writing the token first is how you silently skip a day of deliveries.

Set $top deliberately. SAP Gateway nodes are shared infrastructure; a 50,000-row page is a good way to get your service throttled during month-end close.

Pattern 2: RFC/BAPI for ECC estates

With tSAPConnection + tSAPInput you call a function module and map its export parameters and tables to a Talend schema. Three hard-won rules:

• Always use a custom Z-function module for bulk extracts. Have the ABAP team write one that takes a date/timestamp range and returns a flat, typed table. It performs better than generic reads, it keeps you inside the licence, and it survives upgrades because someone owns it.
• Chunk by business key or date, not by row offset. SAP has no stable ordering guarantee across calls; offset paging will duplicate and drop rows under concurrent updates.
• Map SAP types explicitly. Dates arrive as 00000000 for null, decimals carry sign at the end of the string in some RFCs, and MATNR is often zero-padded to 18 characters. Normalise in a tMap with a routine, once, and reuse it everywhere. The alternative is six months of joins that silently miss 000000000000012345 versus 12345.

Pattern 3: IDocs when SAP should push

If the requirement is "tell us when a sales order changes", stop pulling. Have SAP configure an outbound partner profile that writes IDocs to a file port or an ALE queue, and let Talend pick them up:

• Treat IDoc files as an atomic pickup: move to a processing/ directory first, parse, then move to archive/. Same exactly-once discipline as any SFTP feed.
• Parse the segments with tFileInputPositional against the segment definitions, or convert the IDoc-XML variant with tXMLMap. IDoc-XML is worth asking for; positional parsing of 40 segment types ages badly.
• Key your idempotency on the IDoc control number (DOCNUM), not on a filename or a timestamp. Reprocessing is normal in SAP operations, and you will receive the same document twice.

Landing it: don't model SAP in your staging layer

The most common mistake we clean up is a team trying to be helpful — joining VBAK/VBAP or applying currency conversion inside the Talend job. Don't. Land the extract as-is, typed but unjoined, into a raw schema in Snowflake, BigQuery, Databricks, or whatever the target is. Do the joins and the semantics downstream in dbt, where they are versioned, tested, and reviewable. When SAP changes a field length in an upgrade, you fix one landing schema instead of auditing fifty tMap expressions.

Name the raw tables after the SAP object, keep the original field names (MATNR, WERKS, BUKRS), and add three columns to every row: extract timestamp, source system identifier, and the delta token or IDoc control number it arrived under. That last column is what lets you answer "where did this number come from" eighteen months later.

A pre-flight checklist for SAP work

Before you commit a date to the business:

• Access path agreed and approved in writing (OData, CDS, RFC, IDoc)
• JCo libraries licensed, installed on build and runtime, documented
• Service user created with the right authorisation objects — not a named person's account
• Non-production SAP client available with representative data volume
• Agreement on who writes and owns any Z-function modules
• A defined window: SAP batch, month-end close, and your extract should not collide
• Delta strategy: change timestamp field, change pointers, or CDC — confirmed with a functional consultant, not guessed

If more than two of those are open, the SAP extract is not an engineering task yet. It is a scoping task.

Where this is heading

SAP's own direction pushes integration toward Datasphere and governed OData/CDS consumption rather than table-level reads, and the newer agreements make that explicit. For Talend shops that is mostly good news: HTTP and typed views are easier to build, test, and run in a cloud runtime than JCo native libraries on a JobServer ever were. If you are modernising a Talend estate anyway — onto Talend Studio, Qlik Talend Cloud, or an ELT stack — re-platforming the SAP extracts to OData at the same time removes the one dependency that makes your runtime environment special.

If you have SAP feeds in a Talend estate you are migrating or rebuilding, we can inventory them and tell you honestly which ones port cleanly and which need SAP-side work. Get in touch.