Roelant Vos An expert view on Agile Data Warehousing

ETL generation in SSIS (adult steps) 0

ETL generation in SSIS (adult steps)

 

 Full generation of ETL is the missing component towards the model driven design of the Data Warehouse and I am still pursuing various methods for various ETL suites to add to this concept. Some time ago I looked into ETL generation for Microsoft SSIS using the available DTS libraries (see the post) which really were baby steps. At some point one of my colleagues suggested looking into ‘BIML’ (Business Intelligence Markup Language) and the accompanying...

Handling logical deletes in the Data Warehouse 1

Handling logical deletes in the Data Warehouse

 

 While working on a recent project I had a brief discussion on the implementation of logical deletes. This prompted me to define for once and for all how ETL should handle these occurences. This is, of course, assuming that some CDC mechanism provides the required details or you have the capacity to compare full sets of data to derive the logical deletes yourself. For this purpose I drafted a Design Pattern which should explain this...

Kimball versus Inmon: a peace offer? 5

Kimball versus Inmon: a peace offer?

 

 Lately there were some interesting updates in the ever-existing ‘Kimball versus Inmon’ discussion. This this Bill Inmon wrote an article expressing his views on these things: http://www.b-eye-network.com/view/14115. A quote here sums it up for me: ‘If you want a tactical solution, then Kimball is the way to go. If you want a strategic solution, then Inmon is the way to go.’ All in all I think the Kimball approach is a good one if the goal is a stand...

EAI, ESB and ETL and the ETL generation tool 0

EAI, ESB and ETL and the ETL generation tool

 

 It has been very quiet on the weblog / idea repository since November. This was partly due to the summer holiday period (in Australia) and the fact that I’ve finally picked up the work to develop the ETL generation tool based on the described EDW architecture. Work wise I’ve recently done a lot of strategy work with very little practical exposure to the technologies, and this is likely to continue for some time to come....

Data Vault in Australia 0

Data Vault in Australia

 

 Anyone interested in Data Vault in Australia, please sign up for the official Data Vault User Groups (and specifically the Australian group). Just started, and Data Vault does not have many reference projects in Australia, but that can only mean there is much opportunity to set up a good foundation. The site can be found on www.dvusergroup.com. EDIT: the user groups have been dismantled in favour of the linkedIN discussion groups!!!

0

Why a separate surrogate key table?

 

 A question I often (and recently) get is why you should create a separate table to manage the surrogate keys. This is compared to star- or snowflaked models where the surrogate key (and distribution) is part of the table. For Data Vault the Hub entity is the place where the keys are managed. There is rationale for splitting this process into separate tables: A surrogate key table makes it easy to identify the records in...

Powercenter transaction control with XML/text file targets 0

Powercenter transaction control with XML/text file targets

 

Using the port evaluation of Informatica Powercenter to implement transaction control for text or XML file targets.

Enterprise Architecture seminar thoughts 0

Enterprise Architecture seminar thoughts

 

 This posting is about the most complex object that humankind can come up with; an Enterprise. At least that is what John Zachman states at his one day seminar about Enterprise Architecture (physics 101 – Monday 23rd of August 2010 – Sydney). The first rule for the way of working is defined right at the start: when doing anything with an enterprise or start designing anything for an enterprise; make sure you understand the enterprise...