Joining tables in the Persistent Staging Area
Joining tables in the Persistent Staging Area (PSA) could be a practical solution that avoids downstream complexities. This post explains the pattern to do so.
Tagged: PSA
Getting started with Confluent Kafka as a data professional – part 1
An introduction for data professionals on publishing and consuming Kafka events.
Beyond ETL Generation & DWH Virtualisation – what’s next?
An outlook to the future covering potential ways ETL generation and virtualisation could evolve.
Loading too fast for unique date / time stamps – what to do?
Let’s start by clarifying that this concerns the RDBMS world, not the Hadoop world 😉 It’s a good problem to have – loading data too quickly. So quickly that, even at high precision, multiple changes for the same key end up being inserted with the same Load Date/Time Stamp (LDTS). What happens here? A quick recap: in Data Vault the Load Date/Time Stamp (LDTS, LOAD_DTS, or INSERT_DATETIME) is defined as the moment data is recorded...
Staging and interpreting XMLs, BLOBs and similar
Recently I had a couple of conversations about the ‘staging’ (loading data delta into your environment) of multi-structured datatypes such as JSON, XML (and some blobs). Today I had one of these conversations with my esteemed ex-colleagues Bruce and Glenn which made me think some additional information and considerations about the recent post to parse XML using XQuery would be a good idea. These conversations focused on where the interpretation of XML should happen: storing the unmodified (raw) XML...