Introducing Agnostic Data Labs
Announcing Agnostic Data Labs – the new data solution automation platform to truly implemented data warehouse projects your way!
Tagged: Data Warehouse
Joining tables in the Persistent Staging Area
Joining tables in the Persistent Staging Area (PSA) could be a practical solution that avoids downstream complexities. This post explains the pattern to do so.
A not-so-gentle follow-up on bitemporal data challenges
When delivering data from the integration layer (e.g. a Data Vault model) to the presentation layer (anything, but usually a dimensional model or wide table), a key requirement is re-organising data to the selected ‘business’ timeline for delivery.
During this process, we leave the safety of the assertion (technical) timeline behind and start using the real-world state timeline for delivery. This may create some unexpected results!
Why a delete flag makes sense (and when it doesn’t)
Recording a deleted flag is essential to delivering the correct data, and this post explains why.
Major update for TEAM and Virtual Data Warehouse software
TEAM and VDW code generation software for Data Warehouse automation 1.6 release now fully aligned to generic schema for Data Warehouse automation!
The Virtual Data Warehouse
Over the weekend I have written up a brief overview and ‘thought piece’ of what I mean when I talk about a Virtual Data Warehouse and Data Warehouse Virtualisation. Please have a look at the article here: The Virtual Data Warehouse. A special thanks to Bret Victor for sharing a fascinating presentation on ‘inventing in principle’. This is a concept, or I should rather say a principle, I have been working on for some time now....
Data Warehouse versioning… for virtualisation
Recent discussions around Data Warehouse virtualisation made me realise I forgot to post one of the important requirements: version control. In the various recent presentations this was discussed at length but somehow it didn’t make it to the transcript. Data Warehouse virtualisation needs versioning. Think of it this way – if you can drop and refactor your Data Warehouse based on (the changes in your) metadata then your upstream reports and analytics are very likely...
Virtualising your (Enterprise) Data Warehouse – what do you need?
For a while I have been promoting the concept of defining a Historical (raw) Staging Area / archive to complement the Data Warehouse architecture. A quick recap: the Historical Staging Area is really just an insert-only persistent archive of all original data delta that has been received. One of the great things is that you can deploy this from day one, start capturing changes (data delta) and never have to do an initial load again. In...
Data Vault Role Playing in Links and Link Satellites
Recently I have been involved in some (very lively) discussions related to implementation of ‘role playing’ in Data Vault. In other words: how to handle (model) different types of relationships. Over the years my response has been that from the perspective of Data Vault modelling it doesn’t really matter if you create multiple Link relationships between Hubs / business entities, or if you create a single relationship with multiple Link Satellites each handling a relationship...
Data Vault implementation preparations – fundamental ETL requirements
Prior to working my way through the end-to-end ETL solution for Data Vault certain fundamentals must be in place. The reference architecture is one of them and this is largely documented as part of this site and corresponding Wiki. The other main component from an implementation perspective are the requirements for ETL. As all concepts have their place in the reference architecture for good reasons they also have tight relationships and changes to one concept...