Improvements on DWH automation interface (v1.1)
A new release of the Data Warehouse automation interface has been placed on the Data Warehouse automation Github.
The release has been labelled ‘v.1.1’ and can be found in the releases section. Thanks everyone that has provided input on this!
This is work that has been completed as part of preparing TEAM v.1.6 and VDW 1.6.1, so that these tools can operate fully using the principles of the schema. The class library has been implemented in both TEAM and VDW, and Json files generated by TEAM have been successfully tested.
This is a good example of decoupling the metadata management with ETL generation, also since in VDW 1.6.1 the GUI is created based on the content of the available JSON files.
The main changes are related to refinements on the schema definition (JSON schema definition), the class library and regression test files:
- Generic classification object, applied (as a collection / array) at Data Object Mapping and Data Object level
- Extensions to Data Query, including examples of how to set connectivity information
- Related Data Object collection / array at Data Object Mapping level to support any kind of query or table incorporation into a mapping. For example lookups, joins etc.
- Removed some duplicate elements such as type (replaced by classification).
- Created all elements and descriptions in class library.
- New test files (generated by TEAM) to validate complex logic.
Now on to completing the TEAM and VDW releases, and after this back to applying all of this to ETL generation in my new job.
