Roelant Vos

Data solution design patterns, implementation, and automation

Tagged: BIML

0

ETL / General

Generating data logistics using Biml with the schema for Data Warehouse Automation

This post details how you can connect your own Biml automation framework to the schema for Data Warehouse Automation. A focus on being technology agnostic, and repository-less The schema for Data Warehouse Automation provides a way to separate the storage of your design and code generation (meta) data from how you interact with it and how you apply it to generate data solutions. This (meta)data is something I refer to as design metadata, and it...

2

General

Visual Studio code editor (main screen) blank when using Biml Express – issue and fix

I encountered a very annoying issue the other day, and on the off-chance that anyone else may run into it as well I thought I’d post the fix for it. When showing some SSIS generation code using Biml Express (which is still awesome) to a colleague recently I couldn’t get the Biml files to properly open in Visual Studio using the Biml Express plug-in. The original files were still OK (not corrupted or anything), and...

3

Architecture / Data Vault / ETL

Running SSIS packages continuously without scheduling

No more Batch ETL A few weeks ago I wrote a post about the concept of having continuous execution of ETL individual processes to achieve ‘eventual consistency‘. In that post I made the case to step away from ‘Batch’ execution of ETLs, where related processes are executed as a mini workflow, in favour if fully independent execution of the individual (modular) ETL processes. I have spend some time developing this concept in SQL Server using...

1

General

Biml Express 2017 tests, comments and work-arounds

The new version of Biml Express, the free script-based ETL generation plug-in for Visual Studio provided by Varigence, has been out for a few months. Mid-July 2017 to be precise. However only recently I have been able to find some time to properly regression-test this new release against my library of patterns / scripts. The driver is the upcoming Data Modelling Zone event and Data Vault Implementation & Automation training sessions – better keep up...

1

ETL

Tech tip: making SSIS Project Connections generate correctly using BIML Express

A bit more of a technical view on things today. In order to stay up to date with the latest when it comes to generating ETL for the Microsoft stack (SSIS), I recently upgraded from Visual Studio 2013 with BIDS Helper 1.6.6. to Visual Studio 2015 with BIML Express. And this means a lot of regression testing for years and years of increasingly complex BIML and C# scripts. As it turns out it wasn’t too...

ETL generation in SSIS (adult steps) 0

ETL

ETL generation in SSIS (adult steps)

Full generation of ETL is the missing component towards the model driven design of the Data Warehouse and I am still pursuing various methods for various ETL suites to add to this concept. Some time ago I looked into ETL generation for Microsoft SSIS using the available DTS libraries (see the post) which really were baby steps. At some point one of my colleagues suggested looking into ‘BIML’ (Business Intelligence Markup Language) and the accompanying...