Run the transformation pipeline

Now that we know that our pipeline is sane, we will focus on the execution.

The only thing left to do before executing our transformation pipeline is to define portations to import data to Holium and export data from Holium. In our case we will apply two import portations to our sources and one export portation to our shaper.

The different files that we will use as our imported data can be found in ./assets/sources.

In the first file, avocado.json, we can find all retails sales details for avocado in the US. The second file holds the year we want to conduct our operation on.

To create a new portation the holium portation create sub-command can be used. We will also review the created portations by using the sub-command holium portation list :

$ holium portation create \        --direction toHolium \        --node-type source \        --node-name avocado_sales \        --file-path ./assets/sources/avocado.json \        --file-format jsonnew object created: to:source:avocado_sales
$ holium portation create \        --direction toHolium \        --node-type source \        --node-name selected_year \        --file-path ./assets/sources/selected_year.json \        --file-format jsonnew object created: to:source:selected_year
$ holium portation create \        --direction fromHolium \        --node-type shaper \        --node-name sales_results \        --file-path sales_results.json \        --file-format jsonnew object created: from:shaper:sales_results

As we have now defined all necessary information to have a proper transformation pipeline DAG and data sourcing we can execute it!

To do so, the sub-command holium project run is available:

$ holium project runsuccessfully ran the transformation pipeline
1 successful export(s) during execution:shaper:sales_results → sales_results.json

Now if we open our sales_results.json file we can find the generated json object:

$ cat sales_results.json[[["4046",1709447236.0],["4225",1761051252.0],["4770",142770478.0]]]

In the end, we have created a valid pipeline that can run to produce data based on our connections and transformations.

This is the end of the base tutorial ! If you have any more questions please feel free to open a discussion on the repository or reach out to the community.