Virtual Enterprise Data Warehouse ideas & updates (towards 1.2)
Lately I have had a bit more head space to work on some ideas I find interesting, and these are now intended to culminate into ‘version 1.2’ of the Virtual EDW tool I have been developing. I’ve been using this tool extensively for various Data Warehouses and am generally very happy with it as a quick prototyping tool. But what really starts to play up is the requirement for a physical Data Vault (Integration Layer), as most of the logic is generated based off the data dictionary / catalog.
As it turns out, I’m often more than happy to use the views ‘for real’ but in that case I don’t really want the table to be present either. Only if I want to instantiate the table I want to create it (and populate it).
Another (in my mind) sensible change is to remove the need for Excel (and macros) to populate the automation metadata model – this is now incorporated into the tool.
Lastly, as the Data Vault components are now really solid, it helps to be able to quickly generate some (raw) information / data marts as well!
This is quite a big change. The intention is to write some posts about some of these changes and the adjustments work in more detail. So far I’ve been working on doing this and some other bits and pieces as outlined below:
- Multi-threading; output generation is now processed on separate background threads allowing for a more responsive UI and parallel calculations – much quicker when you generate large Data Warehouses!
- Bug fixing: removed some remaining hard-coding (thanks Douglas Jones and Bas Stiekema for letting me know).
- Versioning; now built in as save and retrieve option using a slider – really neat as this allows you to cycle through versions and see the code change over time.
- Automation Metadata maintenance; the menu option is enabled. This removes the need for Excel to store the mapping metadata and populate the metadata repository. This also allows for more advanced validation and some syntax highlighting.
- Model maintenance; upload the model (or reverse-engineer) to generate views without having Data Vault tables physically available.
- Dimensional model update – navigate the Data Vault model and generate Facts and Dimension views!
- Updates in Satellite view creation – some new logic to cater for those pesky remaining situations (record condensing).
- Proper window resizing; controls now resize properly, allowing more efficient maximising of the windows (just to make it a bit nicer).
Watch this space!