Streaming with Talend: Kafka Jobs, Offsets, and Honest Delivery Guarantees

Most Talend estates we inherit are batch estates. Then a source team publishes to Kafka instead of dropping a file, and suddenly someone has to write a streaming job. This tutorial covers the parts that are not obvious from the component palette: offset semantics, what "exactly-once" really buys you, schema handling, and how to decide whether you need a streaming job at all.

First: do you actually need a streaming job?

A Talend Standard job that runs every two minutes, reads whatever is currently available on the topic, commits offsets, and exits is far easier to operate than a job that runs forever. It restarts cleanly, it fits your existing scheduler, its logs rotate normally, and a bad deploy does not leave a zombie consumer holding partitions.

Choose a long-running streaming job only when you genuinely need sub-minute latency or you are consuming a firehose where per-run startup cost dominates. Everything below applies to both shapes; the micro-batch pattern is at the end.

Consuming: tKafkaInput essentials

The component emits a single byte-array (or string) payload column plus, if you enable it, topic/partition/offset/timestamp metadata. It does not parse your message. Plan on tKafkaInput --> tExtractJSONFields (or tConvertType --> tJavaRow) as the standard opening.

The settings that matter:

• Broker list — always more than one bootstrap broker, from context variables, never hard-coded.
• Consumer group id — this is the identity that owns offsets. One group per logical pipeline. If you copy a job and forget to change the group id, the two jobs will steal partitions from each other and each will see a random half of the data.
• Offset reset (earliest / latest) — only applies when the group has no committed offset. It is not a replay switch; teams misread this constantly. latest on a brand-new group silently skips the backlog.
• Auto-commit offsets — turn it off for anything that writes to a database. See below.
• Stop condition — maximum number of messages and/or a time limit turns a streaming component into a bounded batch read. This is the knob that makes the micro-batch pattern possible.
• Timeout / poll interval — if your downstream processing per poll exceeds max.poll.interval.ms, the broker evicts you from the group mid-run and you get a rebalance storm. Raise the timeout or shrink the batch.

Offsets: the only part that determines correctness

Auto-commit means the consumer periodically tells the broker "I have processed up to offset N" on a timer, independent of whether your tDBOutput actually committed. Crash between the two and you lose records. That is not at-least-once; it is at-most-once with extra steps.

The correct order for a job that loads a database:

1. Poll a bounded batch of messages.
2. Transform and write to the target, in a transaction.
3. Commit the target transaction.
4. Then commit Kafka offsets.

In Talend that means: auto-commit off, tKafkaCommit (or the commit option on the input) wired on an OnSubjobOk link after the tDBCommit. If step 3 fails, the job dies without committing offsets and the next run re-reads the same messages. You have chosen at-least-once, deliberately.

tKafkaInput --Main--> tExtractJSONFields --> tMap --> tDBOutput (commit every: 0)
     |
     OnSubjobOk --> tDBCommit --OnSubjobOk--> tKafkaCommit
     |
     OnSubjobError --> tDBRollback --> tDie("batch failed, offsets not advanced")

Set commit every to 0 on the output component so Talend does not sneak an auto-commit in mid-flow.

"Exactly-once" is idempotency, not a checkbox

Kafka's transactional producer gives exactly-once between Kafka topics. The moment your sink is Snowflake, Postgres, or an API, that guarantee ends. At-least-once delivery plus an idempotent write is the only combination that survives a restart.

Make the write idempotent:

• Derive a deterministic business key from the message — a natural key, or topic|partition|offset if there is nothing better. Never a UUID generated in the job.
• Land raw messages into a staging table keyed on that value, then MERGE into the target on it. Reprocessing the same batch becomes a no-op.
• If the sink is an API, use its idempotency key header. If it has none, keep a small dedupe table of processed keys with a TTL.

A useful rule: if you cannot re-run yesterday's batch twice and get the same target state, you do not have a pipeline, you have a coin flip.

Schema handling and poison messages

One malformed message must not stop the topic forever. Two habits prevent that:

Reject, don't die. Use the reject output of tExtractJSONFields/tMap and route failures to a dead-letter topic or an error table carrying the raw payload, topic, partition, offset, error text, and timestamp. Alert on the rate of rejects, not on the first one.

Pin the contract. If producers use a schema registry with Avro, do not hand-roll deserialization in a tJavaRow. Read the payload as bytes and deserialize with the registry client in a routine, so a producer's incompatible change fails loudly at deserialization rather than quietly mapping a renamed field to null. If producers use plain JSON, treat unknown fields as forward-compatible (ignore them) and missing required fields as a reject.

Producing: tKafkaOutput

Producing is simpler, with three decisions:

• Key. Kafka orders messages only within a partition, and partition is chosen by key hash. If downstream consumers need per-customer ordering, key by customer id. A null key gives you round-robin and no ordering guarantee at all.
• Acks. acks=all for anything that matters. acks=1 will lose messages on a broker failover, and the throughput you gain is rarely the bottleneck in a Talend job.
• Serialization. Emit a stable envelope: event type, version, event timestamp, source system, and the payload. Consumers you have never met will thank you.

Batch producing from a database table? Read with a bounded query, produce, and only then mark rows as sent — same ordering logic as the consumer side, mirrored.

The micro-batch pattern we deploy most often

tPreJob --> load context (brokers, group id, batch size)

tKafkaInput (auto-commit OFF, max messages = 50000, timeout = 30s)
  --Main--> tExtractJSONFields --> tMap --> tDBOutput (staging, commit every 0)
  --Reject--> tDBOutput (dead-letter table)

OnSubjobOk --> tDBRow (MERGE staging INTO target ON business_key)
OnSubjobOk --> tDBCommit
OnSubjobOk --> tKafkaCommit
OnSubjobOk --> tDBRow (INSERT run stats: messages, rejects, max offset, duration)

Scheduled every one to five minutes. Latency is a few minutes, throughput is high because writes are bulk, and failure handling is a re-run rather than a pager. If a run overlaps the next, guard it with a lock row or your scheduler's concurrency setting — two instances of the same consumer group is a rebalance, not parallelism.

To scale, add partitions on the topic and run N instances of the job in the same consumer group; Kafka assigns partitions across them. Running more job instances than partitions just leaves instances idle.

Operating it

Monitor consumer group lag, not job success. A job can succeed every run and still fall further behind forever. Lag per partition, trended, is the single most useful graph. Alert when lag exceeds what your batch size can drain in a few cycles.

Also log, per run: messages consumed, rejects, max offset per partition, and wall-clock duration. When someone asks "did we get the 14:05 event?", that table answers it in one query.

Common failure modes, in the order we usually find them

1. Auto-commit left on, so restarts lose records silently.
2. Two jobs sharing a consumer group id after a copy-paste deploy.
3. latest offset reset on a new group, skipping the backlog nobody noticed was missing.
4. No dead-letter path, so one bad message wedges the topic behind a crash-loop.
5. Non-idempotent writes, so the re-run that "fixed" the outage double-loaded a day of revenue.
6. Long-running streaming jobs on a server with no supervision, discovered dead three weeks later.

None of these are Kafka problems or Talend problems. They are design defaults that need to be chosen on purpose.

---

If you are bolting streaming ingestion onto a batch Talend estate and want the design reviewed before it hits production, get in touch — it is a short conversation that saves a long incident.