Category: General

0

Registration now working!

I’ve finally properly (I think) configured the website to allow registration and the adding of comments in a user-friendly way, without having the burden of endless spambots. Registration, the creation of an account, will allow commenting and discussing content on the site itself which is a big improvement over the current email-based correspondence. After having the account setup you will receive a welcome email and be able to log in to the site using the...

 
0

Adopting GitHub for documentation, and resulting blog changes

After having used Git(Hub) to work and collaborate on code for a long time, I have recently spent some time to merge and move various documentation artefacts to GitHub as well. This covers the Data Integration framework and Enterprise Data Warehouse (EDW) architecture documentation, most importantly the various Design Patterns and Solution Patterns. These patterns form the central body of content that actually try to explain how things work in practice. I think it makes a...

 
0

Is Data Vault becoming obsolete?

What value do we get from having an intermediate hyper-normalised layer? Let me start by stating that a Data Warehouse is a necessary evil at the best of times. In the ideal world, there would be no need for it, as optimal governance and near real-time multidirectional data harmonisation would have created an environment where it is easy to retrieve information without any ambiguity across systems (including its history of changes). Ideally, we would not...

 
1

Some Q&A on Data Warehouse Virtualisation

I receive a fair bit of questions on the Data Warehouse Virtualisation ideas and wanted to respond and discuss this via this post. I don’t have all the answer but can share my views and expectations. When it comes to DWH Virtualisation and the Persistent Staging Area (PSA), the questions generally fall into two categories: Isn’t it too slow? How about performance? Surely users don’t want to wait for hours to see results? Why bother...

 
1

Biml Express 2017 tests, comments and work-arounds

The new version of Biml Express, the free script-based ETL generation plug-in for Visual Studio provided by Varigence, has been out for a few months. Mid-July 2017 to be precise. However only recently I have been able to find some time to properly regression-test this new release against my library of patterns / scripts. The driver is the upcoming Data Modelling Zone event and Data Vault Implementation & Automation training sessions – better keep up...

 
5

Updated sample and metadata models for Data Vault generation and virtualisation

Note: this page has been updated in March 2019 to update QuickDBD to SQL DBM. I have a license to use SQL DBM to generate code, but for viewers the forward engineering will be limited to 10 objects. However, the full model is visible (and can be saved / copied) and the sample code can be generated from the TEAM application if required. If this doesn’t work feel free to email me as well at...

 
0

Updated the Data Vault implementation & automation training for 12-14 June in Germany

On the 12th-14th of June I will be delivering the newly styled and updated Data Vault implementation and automation training together with Doerffler & Partner. I am really looking forward to continue the collaboration after last year’s awesome Data Vault Day (organised by Doerffler as well). Working really hard to wrap up the next layer of virtualisation to discuss there and I’m really excited about it: imagine having multiple versions of not only the Data...

 
2

Advanced row condensing for Data Vault Satellites

When it comes to record condensing, DISTINCT just doesn’t cut it. I’ve been meaning to post about this for ages as the earliest templates (as also posted on this site) were not flexible enough to work in all cases (which is what we strive for). Record condensing Record condensing is making sure that the data delta (differential) you process is a true delta for the specific scope. It’s about making sure no redundant records are processed into...

 
0

Unknown keys (zero keys or ghost keys) in Hubs for Data Vault 2.0

I am still working towards capturing the generation (using BIML in SSIS) and virtualisation (using views / SQL) of the Presentation Layer (in a Dimensional Model). But before we get there, some topics need to be addressed first. One of these is the requirement to have ‘unknown’ keys available in the Hubs. Thankfully, this is one of the easiest concepts to implement. The basic idea is that you create a dummy record in the Hub which...