Roelant Vos An expert view on Agile Data Warehousing

Implementing User Managed data (User Managed Staging Tables) 0

Implementing User Managed data (User Managed Staging Tables)


 Related to the handling of reference data it is sometimes required to feed information into the Data Warehouse that does not have a formal source in the organisation’s information landscape. This information can be needed in the form of reference data (additional information about attributes provided by a source), relationships or really anything that is required to ultimately meet reporting requirements. It can be a vital element to ‘glue’ information together or to provide details...

Data Vault comparisons 1

Data Vault comparisons


 I have drafted a comparison between Data Vault and normalised (3NF) and denormalised (Kimball) models for reference. This comparison is applicable for using these models as the core Data Warehouse model as opposed to modelling for reporting purposes (i.e. data marts).

Data Vault versus the persistent Staging Area 3

Data Vault versus the persistent Staging Area


 One of the questions I regularly get during presentations is what the benefits of Data Vault are over a persistent Staging Area. In other words: why go through the effort of defining a Data Vault model when you can receive the same ‘regeneration / recreation’ capabilities with a persistent Staging Area which directly feeds a Dimensional Model or similar presentation model. First off; in my reference architecture I use both the Data Vault (in the...

Changes to the site(s) 0

Changes to the site(s)


 In preparation of enabling an online (proof of concept) ETL generation site  I have updated various sections of this weblog to conform to the implementations (templates) used for ETL generation. Moreover most documentation has been moved to the corresponding Wiki and consequently removed from this site. Ultimately the Wiki is better suited to add more detailed documentation and examples and as such only the high level concepts and positioning are available on this site. To complete the...

ETL generation in SSIS (adult steps) 0

ETL generation in SSIS (adult steps)


 Full generation of ETL is the missing component towards the model driven design of the Data Warehouse and I am still pursuing various methods for various ETL suites to add to this concept. Some time ago I looked into ETL generation for Microsoft SSIS using the available DTS libraries (see the post) which really were baby steps. At some point one of my colleagues suggested looking into ‘BIML’ (Business Intelligence Markup Language) and the accompanying...

Handling logical deletes in the Data Warehouse 1

Handling logical deletes in the Data Warehouse


 While working on a recent project I had a brief discussion on the implementation of logical deletes. This prompted me to define for once and for all how ETL should handle these occurences. This is, of course, assuming that some CDC mechanism provides the required details or you have the capacity to compare full sets of data to derive the logical deletes yourself. For this purpose I drafted a Design Pattern which should explain this...

Kimball versus Inmon: a peace offer? 5

Kimball versus Inmon: a peace offer?


 Lately there were some interesting updates in the ever-existing ‘Kimball versus Inmon’ discussion. This this Bill Inmon wrote an article expressing his views on these things: A quote here sums it up for me: ‘If you want a tactical solution, then Kimball is the way to go. If you want a strategic solution, then Inmon is the way to go.’ All in all I think the Kimball approach is a good one if the goal is a stand...

EAI, ESB and ETL and the ETL generation tool 0

EAI, ESB and ETL and the ETL generation tool


 It has been very quiet on the weblog / idea repository since November. This was partly due to the summer holiday period (in Australia) and the fact that I’ve finally picked up the work to develop the ETL generation tool based on the described EDW architecture. Work wise I’ve recently done a lot of strategy work with very little practical exposure to the technologies, and this is likely to continue for some time to come....