Roelant Vos An expert view on Agile Data Warehousing

0

Comparisons between Data Warehouse modelling techniques

 

 This post provides an overview of the main pros and cons for various Data Modelling techniques: Third Normal Form (3NF) – The Corporate Data Model. Dimensional Modelling – Facts and Dimensions. Hybrids – Best of both worlds? Data Vault, Anchor Modelling and similar. It has become a bit of a large post but then again, there is a lot of ground to cover. Third Normal Form (3NF) The pros for 3NF are: Most IT professionals...

0

When is Data Vault a suitable solution?

 

 Over the years I have had mixed experiences pitching Data Vault in various situations and have come to a couple of conclusions with regards to applying (or proposing) Data Vault. It is related to various discussions such as having a 2-tiered (layered) or 3-tiered Data Warehouse architecture which in itself is subject to various considerations. And once you decided you do need a 3-tiered approach the discussion between (typically) 3NF and Data Vault as modelling...

Data Vault in Brisbane (Australia): the new user group is up! 0

Data Vault in Brisbane (Australia): the new user group is up!

 

 Happy New Year! We have just initiated a platform for local Brisbane Data Vault enthusiasts to get together and share information and improve the methodology. If you’re living or working close to Brisbane, definitiely check out the Brisbane Data Vault user group. I look forward to meeting anyone interested through this new portal!

Data Vault implementation A-Z: Staging data (the conceptual side) 0

Data Vault implementation A-Z: Staging data (the conceptual side)

 

 This is the first of a planned series of implementation designs for implementing Data Vault in an end-to-end Data Warehouse environment. The positioning of the Data Vault concepts and techniques in the greater design of the system (reference architecture) is documented elsewhere in this site, mainly in the ‘papers’ section. Data Vault in itself does not provide a complete solution for most Data Warehouse purposes but provides a great set of modeling techniques to design the...

Data Vault implementation preparations – fundamental ETL requirements 0

Data Vault implementation preparations – fundamental ETL requirements

 

 Prior to working my way through the end-to-end ETL solution for Data Vault certain fundamentals must be in place. The reference architecture is one of them and this is largely documented as part of this site and corresponding Wiki. The other main component from an implementation perspective are the requirements for ETL. As all concepts have their place in the reference architecture for good reasons they also have tight relationships and changes to one concept...

Designing reference tables for the Data Warehouse 1

Designing reference tables for the Data Warehouse

 

 In a typical Data Warehouse it is common to introduce additional descriptive information that is not provided by the operational systems feeding data into the Data Warehouse. However, the exact positioning and implementation of this reference data can cause confusion. Especially when applying this concept in the Integration Layer. Reference data is additional contextual or descriptive information that is not provided by the source system. Examples are descriptions for industry standard codes. This information can...

Implementing User Managed data (User Managed Staging Tables) 0

Implementing User Managed data (User Managed Staging Tables)

 

 Related to the handling of reference data it is sometimes required to feed information into the Data Warehouse that does not have a formal source in the organisation’s information landscape. This information can be needed in the form of reference data (additional information about attributes provided by a source), relationships or really anything that is required to ultimately meet reporting requirements. It can be a vital element to ‘glue’ information together or to provide details...

Data Vault comparisons 1

Data Vault comparisons

 

 I have drafted a comparison between Data Vault and normalised (3NF) and denormalised (Kimball) models for reference. This comparison is applicable for using these models as the core Data Warehouse model as opposed to modelling for reporting purposes (i.e. data marts).

Data Vault versus the persistent Staging Area 3

Data Vault versus the persistent Staging Area

 

 One of the questions I regularly get during presentations is what the benefits of Data Vault are over a persistent Staging Area. In other words: why go through the effort of defining a Data Vault model when you can receive the same ‘regeneration / recreation’ capabilities with a persistent Staging Area which directly feeds a Dimensional Model or similar presentation model. First off; in my reference architecture I use both the Data Vault (in the...

Changes to the site(s) 0

Changes to the site(s)

 

 In preparation of enabling an online (proof of concept) ETL generation site  I have updated various sections of this weblog to conform to the implementations (templates) used for ETL generation. Moreover most documentation has been moved to the corresponding Wiki and consequently removed from this site. Ultimately the Wiki is better suited to add more detailed documentation and examples and as such only the high level concepts and positioning are available on this site. To complete the...