CSV still moves a surprising amount of data, but most lakehouse and warehouse targets now expect columnar or schema-carrying files. Two formats come up constantly in Talend engagements: Parquet for analytics and Avro for row-oriented exchange and streaming. This tutorial covers how to choose, how to build the jobs, and what usually goes wrong.
Parquet or Avro?
| Need | Better fit |
|---|---|
| Analytical scans, selecting a few columns | Parquet (columnar, compressed, predicate pushdown) |
| Record-at-a-time exchange, Kafka payloads | Avro (row-based, embedded schema) |
| Schema evolution with added optional fields | Both; Avro is more explicit about defaults |
| Loading Snowflake, BigQuery, Redshift, Databricks | Parquet is the usual staging format |
A simple rule: if people will query it, write Parquet. If a system will consume it record by record, write Avro.
Writing Parquet from a Studio job
Talend Studio provides Parquet components in its big data and file families (for example tFileOutputParquet, and the Parquet-capable components for HDFS and cloud storage). Exact component names and availability vary by Studio version and license, so check the palette in your release.
- Build the flow: source input → tMap → Parquet output.
- Set the output schema explicitly. Do not rely on a guessed schema from the first rows.
- Pick a compression codec. Snappy is a sound default; Gzip or Zstd trade CPU for size if your version offers them.
- Write to a staging path (local, S3, GCS, ADLS) and move to the final path only after the job succeeds, so readers never see partial files.
Type mapping that bites
- Decimals: Declare precision and scale deliberately. A Talend BigDecimal with no scale control can produce a Parquet decimal that your warehouse rejects or silently rounds.
- Dates and timestamps: Decide on UTC up front. Mixing local-time Date values with engines that assume UTC shifts data by hours.
- Nulls: Make columns nullable in the schema when the source can be null; a non-nullable column with a null value fails the write.
- Strings with fixed length: Talend schema length is a hint; Parquet does not enforce it. Validate lengths yourself if the target does.
Avro and schemas
Avro files carry their schema, which is the main advantage. Keep the schema (.avsc) in version control and generate or check the Talend schema against it, rather than the reverse. For added fields, give them a default so older readers keep working. Renaming a field is a breaking change; add the new one and deprecate the old.
Partitioning output
Write large datasets as many moderately sized files in partition folders such as event_date=2026-03-01/. Guidelines:
- Partition on a low-cardinality column that queries filter on, usually a date.
- Aim for files in the roughly 128 MB to 1 GB range. Thousands of tiny files hurt every engine.
- Use a loop (tFlowToIterate or a pre-computed list of partitions) so each iteration writes one partition. Rerunning a partition should overwrite that folder, making the job idempotent.
- Keep the partition column out of the file body if your reader derives it from the path, or keep it in both consistently.
Memory and performance
Parquet writers buffer a row group in memory before flushing. Wide schemas with large row groups can exhaust the default JVM heap. If a job dies with an out-of-memory error on write, raise the heap with the JVM arguments in the Run tab, reduce row group size if exposed, or split the output by partition.
Validate before you ship
- Read the file back with a second tool (for example parquet-tools or a DuckDB query) and compare row counts and sums against the source.
- Check the schema, especially decimals and timestamps.
- Confirm the target engine can prune partitions with an actual query plan.
Common pitfalls checklist
- Schema inferred instead of declared
- Decimal scale lost
- Timestamps in mixed time zones
- Too many small files
- Partial files visible to readers during the write
If you are moving a large Talend estate onto lakehouse storage and want a second pair of eyes on file format and partitioning decisions, get in touch.