Secure File Exchange in Talend: SFTP, PGP Decryption, and Pickups That Run Exactly Once

Banks, payers, government agencies, and media partners still move a great deal of data as files on an SFTP server, signed and encrypted with PGP. It is unglamorous, it is not going away, and it is where a surprising number of production incidents come from: a partner rotates a key, a file lands half-written, a job re-reads yesterday's batch, or an unencrypted extract sits in a landing folder for a week.

This tutorial builds a secure file-exchange pattern in Talend Studio that handles all four problems: pull from SFTP, decrypt with PGP, process exactly once, and push an encrypted response back. Everything here works in Talend Studio 8 and in jobs published to Qlik Talend Cloud with a remote engine.

The pattern

tPrejob -> tSSH/context load (keys, paths)
  |
tFTPConnection (SFTP) -> tFTPFileList -> tFTPGet (to ./work/encrypted)
  |
tFileList (*.pgp) -> tJava (decrypt via BouncyCastle routine) -> ./work/clear
  |
tFileInputDelimited -> validation -> target load
  |
tJava (encrypt response) -> tFTPPut -> tFTPRename (remote archive) -> tPostjob

Two rules drive the design:

1. Never process a file in place. Download to a work directory, process, then move to a dated archive. The remote folder is the partner's; the work directory is yours.
2. Never decrypt into a shared or backed-up folder. Clear-text lands in a short-lived directory you delete in tPostjob.

Step 1: Context variables

Create a context group sftp:

Variable	Type	Notes
host	String	partner SFTP hostname
port	Integer	usually 22
user	String
privkey_path	String	path to your OpenSSH private key on the engine
privkey_pass	Password	key passphrase
remote_in	String	e.g. /outbound
remote_archive	String	e.g. /outbound/processed
remote_out	String	e.g. /inbound
work_dir	String	e.g. /data/work/partnerA
pgp_secret_key	String	path to your private key ring (for decrypting inbound)
pgp_secret_pass	Password
pgp_public_key	String	path to the partner's public key (for encrypting outbound)

Load the two passwords at run time from your secrets store rather than from the built-in context — see Secrets Management for Talend Jobs. Key material in the job artifact is the single most common audit finding on file-exchange jobs.

Step 2: Connect with SFTP, not FTPS

In tFTPConnection set:

• Protocol: SFTP
• Authentication method: Public key
• Private key: context.privkey_path, Key passphrase: context.privkey_pass
• Use Socks proxy: only if your network team says so

Two settings people miss:

• Host key checking. Talend's SFTP components use JSch underneath. Turn host-key verification on and point it at a known_hosts file you control. Accepting any host key means a DNS hijack silently redirects your PHI to someone else.
• Connection reuse. Put tFTPConnection in tPrejob and tick Use an existing connection on every downstream FTP component. Opening a session per file is the usual cause of "too many authentication attempts" lockouts on partner servers.

// tJava in tPrejob, before tFTPConnection, if you need a non-default known_hosts
System.setProperty("userKnownHostsFile", context.work_dir + "/known_hosts");

Step 3: Only pick up finished files

A partner writing a 2 GB file directly into /outbound will hand you a truncated read if you grab it mid-write. Three defenses, in order of preference:

1. Trigger files. The partner writes batch_20260310.csv.pgp then batch_20260310.done. You list *.done with tFTPFileList, strip the extension, and fetch only the matching payload.
2. Temp-then-rename. The partner uploads to .tmp and renames. Filter the mask to *.pgp and you are safe, because rename is atomic on the server.
3. Stability check. No cooperation available? Use tFTPFileProperties twice with a tSleep of 30 seconds between, and only accept files whose size and mtime are unchanged.

Wire tFTPFileList to tFTPGet with an Iterate link and set the local directory to context.work_dir + "/encrypted".

Step 4: Decrypt with a BouncyCastle routine

Talend's tFileInputPGP-style components are not available in every edition, and the paid ones vary by release. A small Java routine using BouncyCastle (bcpg-jdk18on / bcprov-jdk18on) is portable, testable, and the same code runs on every engine. Add the two jars with tLibraryLoad or as Maven dependencies in the project, then create a routine PgpUtil:

package routines;

import java.io.*;
import java.security.Security;
import java.util.Iterator;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.openpgp.*;
import org.bouncycastle.openpgp.jcajce.JcaPGPObjectFactory;
import org.bouncycastle.openpgp.operator.jcajce.*;

public class PgpUtil {

    static { Security.addProvider(new BouncyCastleProvider()); }

    /**
     * {talendTypes} String
     * {Category} PgpUtil
     * {param} string("in.pgp") inFile
     * {param} string("out.csv") outFile
     * {param} string("secring.gpg") keyRing
     * {param} string("pass") passphrase
     * {example} PgpUtil.decrypt(in, out, ring, pass)
     */
    public static String decrypt(String inFile, String outFile,
                                 String keyRing, String passphrase) throws Exception {
        try (InputStream in = PGPUtil.getDecoderStream(new FileInputStream(inFile));
             InputStream keyIn = PGPUtil.getDecoderStream(new FileInputStream(keyRing))) {

            JcaPGPObjectFactory f = new JcaPGPObjectFactory(in);
            Object o = f.nextObject();
            PGPEncryptedDataList encList = (o instanceof PGPEncryptedDataList)
                    ? (PGPEncryptedDataList) o : (PGPEncryptedDataList) f.nextObject();

            PGPSecretKeyRingCollection rings = new PGPSecretKeyRingCollection(
                    keyIn, new JcaKeyFingerprintCalculator());

            PGPPrivateKey privKey = null;
            PGPPublicKeyEncryptedData encData = null;
            for (Iterator<PGPEncryptedData> it = encList.getEncryptedDataObjects(); it.hasNext();) {
                PGPPublicKeyEncryptedData d = (PGPPublicKeyEncryptedData) it.next();
                PGPSecretKey sk = rings.getSecretKey(d.getKeyID());
                if (sk != null) {
                    privKey = sk.extractPrivateKey(
                        new JcePBESecretKeyDecryptorBuilder()
                            .setProvider("BC").build(passphrase.toCharArray()));
                    encData = d;
                    break;
                }
            }
            if (privKey == null) {
                throw new IllegalStateException(
                    "No matching private key for this file. Partner encrypted to the wrong key.");
            }

            try (InputStream clear = encData.getDataStream(
                    new JcePublicKeyDataDecryptorFactoryBuilder().setProvider("BC").build(privKey))) {

                Object msg = new JcaPGPObjectFactory(clear).nextObject();
                if (msg instanceof PGPCompressedData) {
                    msg = new JcaPGPObjectFactory(
                            ((PGPCompressedData) msg).getDataStream()).nextObject();
                }
                if (!(msg instanceof PGPLiteralData)) {
                    throw new IllegalStateException("Signed-only or unexpected PGP packet: " + msg);
                }
                try (InputStream lit = ((PGPLiteralData) msg).getInputStream();
                     OutputStream out = new FileOutputStream(outFile)) {
                    byte[] buf = new byte[65536];
                    int n;
                    while ((n = lit.read(buf)) > 0) out.write(buf, 0, n);
                }
            }
            if (encData.isIntegrityProtected() && !encData.verify()) {
                throw new IllegalStateException("PGP integrity check failed for " + inFile);
            }
        }
        return outFile;
    }
}

Call it from a tJava inside the tFileList iteration:

String src = ((String) globalMap.get("tFileList_1_CURRENT_FILEPATH"));
String name = ((String) globalMap.get("tFileList_1_CURRENT_FILE")).replaceAll("\\.(pgp|gpg|asc)$", "");
String dst = context.work_dir + "/clear/" + name;
routines.PgpUtil.decrypt(src, dst, context.pgp_secret_key, context.pgp_secret_pass);
globalMap.put("clear_file", dst);

Two things worth asserting rather than assuming:

• Always check isIntegrityProtected(). Without the modification-detection packet, ciphertext can be tampered with undetected. If a partner's files never carry it, raise it with them in writing.
• If the partner signs, verify the signature. Decryption proves the file was encrypted to you; only signature verification proves who sent it. For a one-pass signature check, iterate the PGPOnePassSignatureList before reading the literal data and call verify() with their public key.

Step 5: Process exactly once

File jobs rerun. They rerun after a crash, after a partner re-posts, and after an operator clicks the wrong thing in Talend Management Console. Make the pickup idempotent with a small control table:

create table etl.file_ledger (
  partner        varchar(64)  not null,
  file_name      varchar(512) not null,
  file_size      bigint       not null,
  sha256         char(64)     not null,
  first_seen_utc timestamp    not null default now(),
  loaded_utc     timestamp,
  status         varchar(16)  not null,
  primary key (partner, file_name)
);

Before decrypting, insert the row with status = 'CLAIMED' using an insert that ignores conflicts (on conflict do nothing in Postgres, MERGE elsewhere) and check the row count. Zero rows inserted means another run already owns that file: skip it. After the target load commits, update to LOADED.

Store the SHA-256 of the encrypted file too. A partner who re-posts a changed file under the same name is a different problem from a duplicate, and the hash is the only thing that tells them apart:

String sha = org.apache.commons.codec.digest.DigestUtils.sha256Hex(
        new java.io.FileInputStream(src));

Step 6: Archive on the remote server

Once the load commits, move the remote file out of the pickup folder with tFTPRename to context.remote_archive + "/" + TalendDate.formatDate("yyyyMMdd", TalendDate.getCurrentDate()) + "/" + fileName. Renaming, rather than deleting, means a partner dispute can be settled from evidence. If the partner's server does not allow writes to a subfolder, keep a local encrypted archive instead — but keep it encrypted, not the decrypted copy.

Step 7: Encrypt the outbound response

The return leg mirrors the inbound one: write the file, encrypt to the partner's public key with ASCII armour and integrity protection on, then tFTPPut to a .tmp name and tFTPRename to the final name so the partner never reads a partial file. The encryption half of PgpUtil uses PGPEncryptedDataGenerator with JcePGPDataEncryptorBuilder(PGPEncryptedData.AES_256).setWithIntegrityPacket(true) and a JcePublicKeyKeyEncryptionMethodGenerator built from the partner's key.

Prefer AES-256, keep compression on (ZIP or ZLIB) — PGP compresses before encrypting, and a 2 GB CSV typically ships as a few hundred MB.

Step 8: Clean up and fail loudly

In tPostjob:

• Delete work_dir/clear recursively with tFileDelete. Decrypted PHI or cardholder data left on an engine disk is a finding waiting to happen, and remote engines in Qlik Talend Cloud are long-lived hosts, not throwaway containers.
• Close the FTP connection with tFTPClose.
• Route tLogCatcher output to your alerting path as described in Error Handling and Observability for Talend Jobs.

A silent zero-file run is the failure mode that costs the most, because nobody notices for days. Count the files you fetched and fail the job when the count is zero on a day a file was expected:

// tJava after the fetch loop
Integer n = (Integer) globalMap.get("tFTPGet_1_NB_FILE");
if (n == null || n == 0) {
    throw new RuntimeException("No files from partner " + context.partner + " on an expected delivery day");
}

Key rotation: the part everyone forgets

PGP keys expire, and partners rotate them with little notice. Two habits prevent the 2 a.m. call:

• Keep both old and new keys in your secret ring during a rotation window. The decrypt loop above already selects the right private key by key ID, so overlapping keys just work.
• Monitor expiry. Run a weekly job that reads each public key's getValidSeconds() and alerts at 30 days out. This is five minutes of work and it is the single highest-value piece of monitoring on a file-exchange estate.

Where this fits

Secure file exchange is rarely the interesting part of a data platform, but for finance, healthcare, and government work it is the contract. Get the pickup atomic, the decryption verified, the processing idempotent, and the clear-text short-lived, and the pattern will run for years with no attention.

If you are standardizing dozens of partner feeds, or moving a file-exchange estate onto Qlik Talend Cloud remote engines, talk to us — it is work we do regularly.