Back to templates
// Template · DataOps
Dynamic CSV Ingestion
Dynamic CSV Ingestion
with Metadata Injection
Load any number of CSV files into a database using a single reusable pipeline. Instead of building one pipeline per table, this template uses Apache Hop's Metadata Injection to detect schema at runtime — so three pipelines handle all your files, regardless of structure.
Technique
Metadata Injection
3 pipelines
DataOps
CSV
PostgreSQL
Dynamic ETL
Apache Hop
Download templates
3 Apache Hop pipelines
Created Feb 28, 2025
Putki 2026.03+
Created Feb 28, 2025
Putki 2026.03+
Pipeline architecture
Loop
get-csv-files
GetFileNames
create-field-table
SelectValues
remove-extension
ReplaceString
remove-numbers
StringOperations
orders-injection
PipelineExecutor
Injection
get-variables
GetVariable
read-file-metadata
FileMetadata
inject-template
MetaInject
Template
read-csv-file
CSVInput
write-log
WriteToLog
write-data
TableOutput
How it works
Instead of building one pipeline per CSV file, this template uses Metadata Injection to handle any number of files with any schema — using just three reusable pipelines. Add new files to the input folder and they get loaded automatically, with no changes to the pipeline.
1
The loop pipeline scans the input folder
The orders-loop pipeline scans a folder for CSV files, extracts the table name from each filename (e.g.
4-orders.csv → orders), and calls the injection pipeline once per file — passing the filename and target table name as parameters.2
The injection pipeline reads the schema and injects it
The orders-injection pipeline receives the filename and table name, reads the CSV file's metadata — column names, data types, field lengths — and injects that schema dynamically into the template pipeline at runtime. No hardcoded field definitions anywhere.
3
The template pipeline loads the data
The orders-template pipeline is intentionally minimal — just a CSV reader and a database writer. Its schema is entirely defined at runtime by the injection step. It reads the file and loads the data into the correct table without ever knowing the columns in advance.
4
Processing order ensures referential integrity
Files are processed in the order defined by their numeric prefix — so parent tables like
customers are always loaded before child tables like orders. Foreign key constraints are never violated.
What you need
·
Apache Hop / Putki
Metadata Injection is part of the core Apache Hop engine, included in all Putki subscriptions.
·
CSV files following the naming convention
Files must follow the pattern
-.csv 1-customers.csv. The table name in the filename must match the target database table name exactly.·
A target database with the tables already created
Configured for PostgreSQL by default, but works with any JDBC-compatible database. The target tables must exist before running the pipeline.
Included files
orders-loop.hpl
orders-injection.hpl
orders-template.hpl
Details
Type
3 pipelines (.hpl)
Technique
Metadata Injection
Target DB
PostgreSQL (any JDBC)
Putki version
2026.03+
Created
February 28, 2025
3 pipelines (.hpl)
Technique
Metadata Injection
Target DB
PostgreSQL (any JDBC)
Putki version
2026.03+
Created
February 28, 2025
When to use this
When you need to load multiple files or tables with different schemas — without writing a separate pipeline for each one.
Scales to
Hundreds or thousands of tables. Add files to the input folder and re-run the loop pipeline — no pipeline changes needed.
Back to templates
// Template · DataOps
Dynamic CSV Ingestion
Dynamic CSV Ingestion
with Metadata Injection
Load any number of CSV files into a database using a single reusable pipeline. Instead of building one pipeline per table, this template uses Apache Hop's Metadata Injection to detect schema at runtime — so three pipelines handle all your files, regardless of structure.
Technique
Metadata Injection
3 pipelines
DataOps
CSV
PostgreSQL
Dynamic ETL
Apache Hop
Download templates
3 Apache Hop pipelines
Created Feb 28, 2025
Putki 2026.03+
Created Feb 28, 2025
Putki 2026.03+
Pipeline architecture
Loop
get-csv-files
GetFileNames
create-field-table
SelectValues
remove-extension
ReplaceString
remove-numbers
StringOperations
orders-injection
PipelineExecutor
Injection
get-variables
GetVariable
read-file-metadata
FileMetadata
inject-template
MetaInject
Template
read-csv-file
CSVInput
write-log
WriteToLog
write-data
TableOutput
How it works
Instead of building one pipeline per CSV file, this template uses Metadata Injection to handle any number of files with any schema — using just three reusable pipelines. Add new files to the input folder and they get loaded automatically, with no changes to the pipeline.
1
The loop pipeline scans the input folder
The orders-loop pipeline scans a folder for CSV files, extracts the table name from each filename (e.g.
4-orders.csv → orders), and calls the injection pipeline once per file — passing the filename and target table name as parameters.2
The injection pipeline reads the schema and injects it
The orders-injection pipeline receives the filename and table name, reads the CSV file's metadata — column names, data types, field lengths — and injects that schema dynamically into the template pipeline at runtime. No hardcoded field definitions anywhere.
3
The template pipeline loads the data
The orders-template pipeline is intentionally minimal — just a CSV reader and a database writer. Its schema is entirely defined at runtime by the injection step. It reads the file and loads the data into the correct table without ever knowing the columns in advance.
4
Processing order ensures referential integrity
Files are processed in the order defined by their numeric prefix — so parent tables like
customers are always loaded before child tables like orders. Foreign key constraints are never violated.
What you need
·
Apache Hop / Putki
Metadata Injection is part of the core Apache Hop engine, included in all Putki subscriptions.
·
CSV files following the naming convention
Files must follow the pattern
<order>-<tablename>.csv — e.g. 1-customers.csv. The table name in the filename must match the target database table name exactly.·
A target database with the tables already created
Configured for PostgreSQL by default, but works with any JDBC-compatible database. The target tables must exist before running the pipeline.
Included files
orders-loop.hpl
orders-injection.hpl
orders-template.hpl
Details
Type
3 pipelines (.hpl)
Technique
Metadata Injection
Target DB
PostgreSQL (any JDBC)
Putki version
2026.03+
Created
February 28, 2025
3 pipelines (.hpl)
Technique
Metadata Injection
Target DB
PostgreSQL (any JDBC)
Putki version
2026.03+
Created
February 28, 2025
When to use this
When you need to load multiple files or tables with different schemas — without writing a separate pipeline for each one.
Scales to
Hundreds or thousands of tables. Add files to the input folder and re-run the loop pipeline — no pipeline changes needed.