Work progressing on the generic interface for Data Warehouse Automation Metadata
Following the collaboration proposal on a common metadata interface for Data Warehouse Automation there have been been various really good conversations on LinkedIn, email and Skype. There has been some very useful feedback, and I’ve spent some collating this on the Github: https://github.com/RoelantVos/Data_Warehouse_Automation_Metadata_Interface .
This Github now has additional documentation explaining the (WIP) interface format as a JSON schema and a few examples using this schema to define the metadata for source-to-target mappings using this schema. There also is a checklist to make sure the known scenarios can be covered and regression-tested.
Based on this, and because the ‘proof of the pudding is in the eating’, I created and uploaded a working example of how the proposed generic interface format can be used to define metadata and validated, consumed into a C# class and accessed for ETL generation using Handlebars. The documentation & sample code for this have also been uploaded to the Github.
Have a look!
Fleshing out the details should be easy from here on. It doesn’t take a lot of effort to update TEAM (the metadata management tool I’m contributing to) so that it can output metadata that corresponds with this interface schema. Also, with a working example it is easy to extend the C# class to consume whatever we would like to add or change to the JSON schema.
The uploaded example also has some code that loads (‘deserialises’) the various different nested objects and arrays from the JSON file, so they can be access in Handlebars. I have used the JSON.net (Newtonsoft) libraries for this.
As a reminder; the idea is (still) to define a generic source-to-target mapping interface which is independent from the many proprietary solutions (such as TEAM in my case but most people I know have, or are working on, something like this). When the schema is complete perhaps it is easier to exchange metadata this way and generally make the tooling ecosystem more interoperable.
I’ll probably spend a bit more time a bit later on this to wrap up the example code so that it is 100% the same as the previous Handlebars example, and publish a post about it in the ‘simple automation series’. I also created a new page in the site menu to log releases when they become available.
Please reach out to me or via the Github for comments, feedback and suggestions in general!
