Roelant Vos

Data solution design patterns, implementation, and automation

Tagged: Code generation

0

General

The new version of the data automation metadata schema is ready!

The latest version of the schema for data solution automation has been published on Github. This comprehensive collection includes examples, functions, and definitions crucial for metadata management in data solutions, particularly data warehouse systems. You can find the details, code, and documentation here: What has changed? One significant update focuses on the Business Key Definition. Initially, it featured a property called ‘Business Key Component Mappings,’ which listed data item mappings for various business key components...

Delta Lake 0

General

Investigating code generation for Delta Lake using Azure Data Factory

Introduction – back in code generation mode Things have been busy after joining Varigence a few months ago. A major focus has been to develop new code generation features, to enable the BimlFlex data warehouse automation platform to use Azure Data Factory (ADF) Mapping Data Flows (‘Data Flows’)- in particular to interfacing with Delta Lake storage. Basically, to make sure that we can use BimlScript to generate Delta Lake compliant Data Flows and integrate this...