Category: General

UPDATE 2022 – please note that the Load Patterns repository has been deprecated. The code is now managed via the VDW repository! The recent new release of the Virtual Data Warehouse software (VDW) introduces the Handlebars templating engine to generate code, providing much greater flexibility than before. It is now possible to customise the application to a large extent, both by changing the way the metadata is made available but of course also the patterns...

TEAM and VEDW have been upgraded, to v1.5.5.2 and v1.6 respectively! This release marks a major milestone in the development of the Data Warehouse Automation ecosystem. This update requires a repository upgrade and is not backwards compatible. Release notes The changes for TEAM are as follows: Overall improved of error handling and reporting. Meaningful error messages are now reported back if things go wrong, and the error log works at all times, including when creating...

I would like to provide a brief update on the various things that are in development within the Data Warehouse Automation / ETL generation ecosystem of open source tooling. Instead of working on new articles I have spent most of my time pushing some of the bigger development changes. This is what is currently in progress and can be expected in the short term: The Data Integration Framework and DIRECT repositories are being finalised for...

Next training session for the Virtual DWH implementation & automation on 28-30 October in Utrecht, The Netherlands.

Following the collaboration proposal on a common metadata interface for Data Warehouse Automation there have been been various really good conversations on LinkedIn, email and Skype. There has been some very useful feedback, and I’ve spent some collating this on the Github: https://github.com/RoelantVos/Data_Warehouse_Automation_Metadata_Interface . This Github now has additional documentation explaining the (WIP) interface format as a JSON schema and a few examples using this schema to define the metadata for source-to-target mappings using this...

Pretty much everyone I know in the Data Warehouse / Business Intelligence industry has, or has worked on, some kind of metadata management model. This may have been developed to explore smarter ways of development or maintenance of documentation. It could be because of disappointment about vendor / software capabilities, a desire to store certain designs or structure and of course to enable code generation – including ETL. This is innovative work, and has led...