Incorporating ETL generation in DevOps pipelines
If you are interested in code generation and automation for your data solution, it is worth having a look at the Virtual Data Warehouse software to understand how design metadata can be applied to patterns to generate many different outcomes.
In the ecosystem for Data Warehouse Automation as outlined in this blog, the Virtual Data Warehouse can be positioned as the ‘pattern development tool’ where you can modify, test, manage the various code generation patterns needed for a particular deployment into a specific platform and/or ETL environment (‘technical environment’).
The ecosystem separates design and metadata management (‘TEAM‘ and ‘TEAM_graph‘) from the storage of metadata (‘schema for Data Warehouse Automation‘) and application to patterns (‘Virtual Data Warehouse‘).
However, the Virtual Data Warehouse is more of a design tool and is not suited to directly integrate into a DevOps pipeline. Although you can generate into a database to view results of your design activities, if you want to generate and/or deploy code automatically to a target server it is more convenient to use command-line tooling.
This is why the RunDwhAutomation command line utility was created recently. I have created a short video how this utility can be added to a Jenkins pipeline to automatically generate and deploy ETL code (in SQL) as part of an automated process.
In the video, I generate SQL from the Virtual Data Warehouse example metadata and patterns and deploy this to the target database server using Jenkins.
The video is available here: https://www.youtube.com/watch?v=ky7jB5f4upg.
