Areas of collaboration
The following diagram provides an overview of areas that are being addressed on an ongoing basis. If you are interested feel free to reach out at email@example.com. For now most collaborations are limited open source until we’re certain all IP has been sorted out properly.
The intent of sharing this code is to foster increased meritocracy in the BI/DWH community and generally work (together) on something that can be combined using agreed APIs.
The idea is that various people / teams can chase their passion while knowing the work will fit in somewhere in the overall scope. This overview may drastically change over time, as will the composition and scope of the projects – but that’s the nature of the work.
Data Integration Framework
The Data Integration Framework provides the context of the collaboration, as well as an overview of the options & considerations across the various areas and layers.
Contents / functionality of the Github:
- Overview and context for Enterprise Data Warehouse
- Design Patterns (conceptual how-to’s)
- Solution Patterns (implementation in specific technical context i.e. tooling and environments)
ETL Control Framework (DIRECT)
The Data Integration Run-time Execution Control Tool (DIRECT) is a generic execution and control framework that orchestrates the execution of ETL processes. It provides various hooks into an ETL process to manage topics such as restartability, recovery from failure, logging, ETL classification and event handling.
There are many ETL control frameworks. They should be considered as a commodity as they are needed in every project. Let’s make this the best one!
Metadata Management (TEAM)
The Taxonomy for ETL Automation Metadata (TEAM) is a management tool for Data Vault metadata, a component also integrated in the VEDW software.
It offers metadata mapping validation, data entry and visualisation. The metadata within TEAM is used to generate ETL (i.e. using Biml, SQL) using the interface / APIs.
Virtual Enterprise Data Warehouse (VEDW)
As per version 22.214.171.124 VEDW is now separated from TEAM components. This means VEDW is the generation engine and all metadata management is done in TEAM.