Category: General


Virtualising your (Enterprise) Data Warehouse – what do you need?

For a while I have been promoting the concept of defining a Historical (raw) Staging Area / archive to complement the Data Warehouse architecture. A quick recap: the Historical Staging Area is really just an insert-only persistent archive of all original data delta that has been received. One of the great things is that you can deploy this from day one, start capturing changes (data delta) and never have to do an initial load again. In...


Data Visualisation, Data Warehousing and Big Data: one pitch to rule them all

Of the concepts that have emerged over the last few years, the ‘Data Lake’ is not one of my favourites. Although it has to be said I had a lot of fun out of various parodies on Data Lakes – which I’ll not repeat here! While I am on board with the cheap redundant storage concept it is clear that data management is still needed in this day and age (more than ever, really) and that concepts...


Using an ETL platform for your Data Warehouse, is it still relevant?

When I started my career in Data Warehousing and Business Intelligence 15 years ago there was a massive push towards adopting ETL software. Traditionally, specialised ETL software such as Informatica Powercenter, Oracle Warehouse Builder (later superseded by Oracle Data Integration), Microsoft DTS (later superseded by SSIS) and many similar platforms were very successful because of two main reason: ETL software provided a way to ‘explain’ or document what was happening in a way that made...


Data Modeling Zone Europe 2014 – #DMZone

The site of the European Data Modeling Zone (DMZ) is up, and it looks really good! I look forward to being in Europe again for this event, which is hosted in Hamburg on the 29th and 30th of September 2014. On behalf of Analytics8 I will present the ‘model driven design’ approach of how the metadata that is embedded in the data model can be used to forward engineer ETL in various platforms. Feel free...


Data Vault 2.0 – how to handle Referential Integrity?

I was working on adding some of the automation code to support Data Vault 2.0 and this got me thinking about Referential Integrity (RI)  related to the modifications that Data Vault 2.0 requires. With Data Vault ‘1.0’ Referential Integrity is always enabled (except for very big systems – let’s leave that one out of the scope for now – see this older post) and in Data Vault 2.0 this hasn’t changed according to the specifications. For Data...


Data Vault 2.0 – Introduction and (technical) differences with 1.0

While finalizing the content covering Data Vault implementation it is time to start looking forward towards Data Vault 2.0. For this purpose it makes sense to provide an overview of the changes between the two data modelling approaches. To limit the scope to implementation I’ll consider it sufficient to mention that Data Vault 2.0 (DV2.0) is a complete approach covering not only the modelling (that was already part of DV1.0) but also the layered DWH...


Takeaways from the World Wide Data Vault Conference

There were some really great sessions at the World Wide Data Vault Conference (#WWDVC), and I really need to take a bit of time to upgrade some of my templates to DV2.0. Thankfully that’s pretty straightforward with the framework in place already; the changes, or upgrade, can be automated as well using the same metadata. My takeaways are; There is a push towards virtualisation for the Presentation Layer / Dimensional Model (or similar); using views directly...


User Defined Properties for ETL Automation – the final piece of the puzzle

At the end of the development efforts to support true Model Driven Design there are some elements you just don’t want to store in another metadata table. But somehow you need specific information which isn’t available in the Data Dictionary either (or directly derivable that way). In my case I wanted to label attributes within a Dimensional Model to behave following specific paradigms for presenting history – Type0, Type1 or Type2 – and essentially use this information...


Referential Integrity in Data Vault

Traditionally, Referential Integrity (RI) is not enforced for a Data Warehouse in some approaches. This may result that the Foreign Key (FK) constraint is either not generated at all, or generated but disabled on database level. The latter can improve the efficiency of the RDBMS optimiser depending on the platform used (and should by a case-by-case consideration). Unlike operational (transactional) systems, data for a DWH is prepared (scrubbed) before it is inserted and this happens in...


World Wide Data Vault Consortium (User Group) meeting

Hi everyone, I will attend the first World Wide DV Consortium (User Group) meeting in St. Albans (Vermont / USA) and talk about the Data Vault automation approaches in general, but SSIS in particular. Looking forward to it and hopefully with so many Data Vault experts we’ll get the opportunity to build & expand on this. Hope you see you there Roelant