Roelant Vos

Data solution design patterns, implementation, and automation

FAQ – Virtual Data Warehouse

In a JSON mapping file (according to the data warehouse automation schema), what elements are required for VDW to work?

VDW uses the mapping name, the classification (at Data Object level) to generate the screens. If this information is not available, this is defaulted to the name of the target (for the mapping name) and ‘Miscellaneous’ for the classification. 

What templating engine does VDW use?

VDW uses Handlebars (Handlebars.Net) as a templating engine.

Are the patterns all using standard Handlebars functions?

To support all current use-cases, a small number of additional functions have been added to the standard Handlebars library. These are called ‘helpers’, and Handlebars allows custom extensions such as these to be added. 

The standard functionality is used as much as possible, but the following extensions / helpers have been added:

  • {{replicate <int>}} to perform a certain (block of) commands a number of times.
  • {{#stringcompare}} to perform string value evaluation and take next steps accordingly.
  • {{#stringdiff}} the opposite of the sting value evaluation, do something explicitly when values do not match.
  • {{now}} to display the current date/time stamp.
  • {{randomstring}}, {{randomdate}} and {{randomnumber}} to produce randomised values.
  • {{stringwrap}}, which provides a character to ‘wrap’ around the input string
  • {{space}}, adding a determined number of spaces

The link including the helper definitions, including code comments is here: https://github.com/data-solution-automation-engine/data-warehouse-automation-metadata-schema/blob/master/ClassLibrary/DataWarehouseAutomation/DataWarehouseAutomation/HandleBarsHelpers.cs

Do I need two separate applications (TEAM and VDW) to generate ETL?

A: Throughout its evolution, TEAM and VDW have been separated to achieve greater separation and interoperability between their distinctive functions (metadata management and code generation).

We consider managing the metadata a separate function than ETL generation, and have split the tools accordingly. This means that it will be easier for everyone to use their own flavour of either metadata management or code generation tooling, but still collaborate on the metadata schemas and patterns.

At present there are various ways to generate ETL (Biml, VEDW, Handlebars and plain SQL) that all use the TEAM interface to produce Json files. At the same time, VDW uses Json files as input that can be created via different approaches.

Can I get access to the source code?

In the collaboration section of this site an overview is provided of the ‘Data Warehouse Automation tooling ecosystem’ to which VDW belongs. These are all arranged through Githubs, most of which are public. If you would like to access a private repository please reach out to me directly, there are various reasons the private settings is enabled but none that would prevent collaboration with like-minded individuals.