Over time, I recorded some videos to explain some concepts in more detail and published these on YouTube.
Publicly published content
- How to agree to disagree (on Data Warehouse Automation). A short presentation covering the relevance of the schema for Data Warehouse Automation from Knowledge Gap 2021.
Conceptual content
- Eventual consistency: Data Vault ETL without loading dependencies. Implementing Data Vault ETL without any loading dependencies, while still achieve (eventual) consistency. Overview of the involved concepts and frameworks, and including a demonstration of application in a DevOps pipeline.
- Integration Code Generation in DevOps. A brief example of how the RunDataWarehouseAutomation command line utility can be used to generate and deploy code (ETL) in a DevOps context.
- Generating ETL / code from the command line. A short video showing how a lightweight command line utility can help generating ETL / code in a toolchain. Part of the ‘virtual data warehouse’ ecosystem.
- Virtual Data Warehouse – replacing software functionality with patterns. Examples of using patterns and templating to replace built-in functionality for creating Point-in-Time tables, test data and Referential Integrity checks in a Data Warehouse context.
- Confluent Kafka for data professionals. This video provides a brief demonstration supporting the article series on ‘Confluent Kafka for data professionals‘ .The demonstration shows how to create a pub/sub process using a custom object (class). The intent is to explain how the collection of data points can be influenced (in code) to support reporting and analysis in a Data Warehouse / Business Intelligence context.
TEAM and VDW specific content
- Installing TEAM and Virtual Data Warehouse. A short introduction on the basic installation and configuration of the TEAM and Virtual Data Warehouse application. This explanation shows the minimal step to generate some Data Warehouse code. For a more complete example, please try switching the repository type (in the configuration screen) to ‘SQL Server’ and re-create the samples. This will also create the (physical) structures in the database, which allows you to use the ‘generate in database’ option in Virtual Data Warehouse.
- Using TEAM and Virtual Data Warehouse to generate a Data Warehouse. Create a simple Virtual Data Warehouse using the example metadata, and switching this solution to a physical (materialised) version using the available code generation patterns.
- TEAM Graph Part 1 – using DGML. This video explains the thinking behind the graph representation in TEAM. For a long time, TEAM has the functionality to export metadata as a Directed Graph Markup Language (DGML) file. DGML is a great way to visualise the relationships between data objects and attributes, and is useful to explain the way the model can be shown at a more abstract level. In the video I show how TEAM derived ‘subject areas’ by grouping sets of source-target mappings together in memory, and that from a modelling point of view this is an easy stepping stone into moving away from a (too) physical representation. The step towards automatically refactoring the (now underlying) physical model can be simulated by moving attributes between entities within a subject area – a reference to The Engine concept).
- TEAM Graph Part 2 – graphs using yFiles. This second video on graph representation for Data Warehouse automation metadata focuses on the yFiles (class)library from yWorks – a company specialised in all things graph. yFiles provides interactivity (event handling) which allows for coding behaviour for graphs in the application.
- Improvements implemented in TEAM release 1.6.1. The new version of TEAM, the Taxonomy of ETL Automation Metadata, has introduced the concepts of ‘environments’ and ‘connections’ to make it easier to apply the data solution design to different technical environments. This video contains a short overview of changes made, and why these help to deliver data solutions in a better way.