Using agile BI is a sure way to deliver useful information to the business fast enough to keep ahead of the rapid changes in the marketplace, but the data warehousing part has to fit into a carefully architected blueprint to ensure the longevity and cost-effectiveness of the implementation.
Implementing agile BI is surprisingly similar to building massive Sodor railways on the floor with my two toddlers… adapting to ever-changing whims of what they want to build where, playing with parts of the track still under construction and micro-managing them to construct their parts of the track roughly in line with the intended layout. And you have to react positively (and quickly) to their frequent joyful cries of “track maintenance!” when their clumsy little feet knock over the bridges and rail lines…
To keep up with changes in the marketplace, business frequently and rapidly alters their information requirements and usage patterns. Agile BI is the only real practical way to deliver information of value to the business fast enough.
However, the most complex of agile BI is “agile data warehousing”, where parts of the data warehouse must be designed, implemented and populated in quick timeframes, while sticking to a blueprint model. I normally prefer having a 80% straw model in place before even starting to fill in the details during each sprint. To do this you need a good understanding of the business and the data representing the business processes. Just-in-time data modeling gets very tricky if you do not know the subject areas and where they integrate.
You have to get the conformed dimensions right from the start. The concept of one single instance of each key entity occurrence on the correct level of detail is crucial, regardless of which model you use. This is even more important with agile data warehousing because the parts of the data warehouse have to be dynamically slotted around these key dimensions as you finalise and populate them during each sprint. With agile implementation cycles you have even less time to rework and repopulate a data model if you had the conformed entities wrong.
So how do you draw up a straw model? In my latest project (in healthcare) there was so much information available! In addition to studying the tables and columns used by the source systems and having informal discussions with the users, there is a wealth of information on the web. BI tool vendors proudly boast what KPIs their packages provide, healthcare specialists discuss what should be tracked and if you search wide enough you find even more detailed information. If you analyse these inputs together as you would the requirements
from many users, you can draw up a good baseline model. At least you will get the key conformed dimensions with their key attributes identified. The really fine detail can be fleshed out during the sprints.
In the next instalment on agile data warehousing, I will discuss how to fill the Fat Controller’s carriages… that is, how to populate the data warehouse when deploying agile BI.
