Roelant Vos An expert view on Agile Data Warehousing

1

Data Vault ETL Implementation using SSIS: Step 3 – Hub ETL – part 1 – overview

 

 The Staging Area and Historical Staging, both part of the conceptual Staging Layer, are not directly related to Data Vault although at least the Staging Area ETL paves to way with the definition of the Event Date/Time. The Hub however is the first true Data Vault template to be implemented in SSIS. Please note that this is true for DV1.0; for DV2.0 the Staging Area is incorporated into the design a bit more. The Hub...

2

Data Vault ETL Implementation using SSIS: Step 2 – Historical Staging ETL

 

 The archiving process such as the Historical Staging ETL are also not conceptually part of Data Vault but can play an important factor in the complete Enterprise Data Warehouse (EDW). While the Staging Area directly supports the Data Vault message the Historical Staging is really optional and does not impact Data Vault processes. For the sake of completion the ETL overview for the Historical Staging is provided regardless. As with any ETL process within the...

0

Data Vault ETL Implementation using SSIS: Step 1 – Staging Area ETL

 

 While technically (and conceptually) not really part of Data Vault the first step of the Enterprise Data Warehouse is to properly source, or stage, the data. This is a critical step, and often one of the most difficult to get right. It also has direct impact on the core DWH / Data Vault development, or at least supports its core message and principles. This impact can be summarised by the definition that the purpose of...

Data Vault evening event – Brisbane – SSIS automation for Data Vault 0

Data Vault evening event – Brisbane – SSIS automation for Data Vault

 

 For everyone available and interested (and in the neighbourhood) a new session has been planned for the Brisbane DWH and Data Vault modelling interest group. It’s all about using SSIS to automate your DWH development. The session is hosted on the 6th of June 2013, from 17:30 onwards. More information is here: http://www.meetup.com/Data-Vault-and-DWH-modeling-interest-group-Brisbane/events/118243642/

0

Comparisons between Data Warehouse modelling techniques

 

 This post provides an overview of the main pros and cons for various Data Modelling techniques: Third Normal Form (3NF) – The Corporate Data Model. Dimensional Modelling – Facts and Dimensions. Hybrids – Best of both worlds? Data Vault, Anchor Modelling and similar. It has become a bit of a large post but then again, there is a lot of ground to cover. Third Normal Form (3NF) The pros for 3NF are: Most IT professionals...

0

When is Data Vault a suitable solution?

 

 Over the years I have had mixed experiences pitching Data Vault in various situations and have come to a couple of conclusions with regards to applying (or proposing) Data Vault. It is related to various discussions such as having a 2-tiered (layered) or 3-tiered Data Warehouse architecture which in itself is subject to various considerations. And once you decided you do need a 3-tiered approach the discussion between (typically) 3NF and Data Vault as modelling...

Data Vault in Brisbane (Australia): the new user group is up! 0

Data Vault in Brisbane (Australia): the new user group is up!

 

 Happy New Year! We have just initiated a platform for local Brisbane Data Vault enthusiasts to get together and share information and improve the methodology. If you’re living or working close to Brisbane, definitiely check out the Brisbane Data Vault user group. I look forward to meeting anyone interested through this new portal!

Data Vault implementation A-Z: Staging data (the conceptual side) 0

Data Vault implementation A-Z: Staging data (the conceptual side)

 

 This is the first of a planned series of implementation designs for implementing Data Vault in an end-to-end Data Warehouse environment. The positioning of the Data Vault concepts and techniques in the greater design of the system (reference architecture) is documented elsewhere in this site, mainly in the ‘papers’ section. Data Vault in itself does not provide a complete solution for most Data Warehouse purposes but provides a great set of modeling techniques to design the...

Data Vault implementation preparations – fundamental ETL requirements 0

Data Vault implementation preparations – fundamental ETL requirements

 

 Prior to working my way through the end-to-end ETL solution for Data Vault certain fundamentals must be in place. The reference architecture is one of them and this is largely documented as part of this site and corresponding Wiki. The other main component from an implementation perspective are the requirements for ETL. As all concepts have their place in the reference architecture for good reasons they also have tight relationships and changes to one concept...

Designing reference tables for the Data Warehouse 1

Designing reference tables for the Data Warehouse

 

 In a typical Data Warehouse it is common to introduce additional descriptive information that is not provided by the operational systems feeding data into the Data Warehouse. However, the exact positioning and implementation of this reference data can cause confusion. Especially when applying this concept in the Integration Layer. Reference data is additional contextual or descriptive information that is not provided by the source system. Examples are descriptions for industry standard codes. This information can...