Last week I presented a paper on the Impact of Big Data on Data Warehousing at the ARK Group’s conference on Revolutionising the Data Warehouse and Business Analytics, which was held in Melbourne. In this post I give you my impressions of the conference highlights and share with you some of my insights.
Alan Duncan, of SMS Management and Technology set the scene by stating that the challenges we are now facing include not only the volume of data, but also that decision makers need all the data, all the time. New types of data as well as new business problems require that we now look at newer forms of analytics to give us answers, and in new record times I might add. This in turn adds its share of organisational challenges, especially on IT to meet the demands. I would also add the challenge of the organisation’s appetite and ability to act and react quickly enough on the additional insight gained
this way. Alan concluded that handling the additional foresight, as provided through more widely applied predictive analytics, is especially a new challenge for many organisations. This I agree with, as many organisations have not yet added the future view and an analysis of forward looking measurements to their decision-making processes. Herein lies the actual challenge, because organisations have to change their decision-making processes from being budget and target driven to a more predictive model.
Robert Hillard of Deloittes discussed the Business Case of Data Warehousing and Business Analytics. This is such an important topic as the value of BI to the business is so often questioned – especially for older, more cumbersome and inefficient implementations. He gave a categorisation of the value of information – it can be considered from various angles, namely intrinsic value, business value, loss value, (increased) performance value, economic value, and market value. For each of these a different estimation is used, and it has a totally different meaning to the business. Note that he used the word estimation, not calculation. As I have also many times said to clients, the real value and the real ROI is almost impossible to calculate, and instead of taking man-months to do a probably inaccurate calculation, you can work with a very well thought through and probably more reflective estimation.
Russell Garnett of Asciano gave an overview of their executive supported “data first” project, where they managed to get the executive mandate to get all the core data into the data warehouse first, then build a semantic layer around it to make it more business accessible, and then from there they can very quickly deliver it to the business. Not many BI competency managers are in a position to have such executive support – in most cases we have to deliver insight to the business at break-neck speed, while getting the data into the data warehouse in the background. Once Asciano’s team have sourced all the data, they should be in a position to deliver information and insight to the business with high levels of agility. This mandate, of course, is a little pie in the sky for most BI competencies, especially if they are not truly supported at the executive level. Then you have to work on the quick wins to deliver value to the business in order to gain that support.
Robbert Barnfather gave an excellent presentation on Maturing the Organisation’s Data Quality Capability, illustrating the approach and framework they used to improve address quality at Australia Post. Two insightful statements Robbert made that I want to share are 1) Data Quality is mostly about people – they cause all the problems and they are crucial to solving them, and 2) For a data quality initiative to really work, you need to encompass system change, organisational change, and data quality governance in one initiative. He also illustrated how applicable the Data Governance Institute’s data governance framework is for setting up, implementing and tracking a data quality initiative.
Brenda Ryan of UnitingCare Health illustrated how they reduced an enormous amount of “Biological ETL” (i.e. manual spreadmart consolidation) through an agile approach by a very small team where they integrated data from two sources to deliver very valuable insights to the business. Interestingly enough, the managers of the other data warehouses (yes, there are many) are very supportive of her group’s work, as it takes the pressures of quick delivery off their teams. Despite the successful quick win, I question the sustainability of this approach and whether it doesn’t create merely another duplication of data in the organisation?
Papiya Chakravati gave a very entertaining presentation on how they use totally independent data marts for delivering business analytics very responsively to their customers at the Australian Sports Commission. It is astonishing how many non-integrated data sources they have to deal with. Getting that all integrated into an EDW would take forever, but it will be even more interesting to see what happens when the set of independent data marts exceed that manageable threshold. Another successful quick win, not necessarily sustainable?
Dr Neil Fraser of Macquarie University gave a very insightful and stimulating presentation on innovative and cost-effective approaches for business analytics. He covered Big Data, cloud computing, new types of analytics and open source platforms based on the experience he has gained through managing the structured data as well as the mass of unstructured and multi-structured data at the university.
During the panel discussion at the end of the first day, Robert Hillard and Peter Missen (of Telstra) addressed various topics, from a view of the employee’s work area as it is expected to be in 2020, through to the BI wish list of today/tomorrow, covering the impact of Big Data on organisations today, the viability of the cloud, and many more stimulating discussion points.
The second day I attended the workshop on anomaly detection and risk analysis by professor Longbing Cao of the University of Technology Sydney, where they do very interesting industry-partnered research at the Institute of Advanced Analytics. Anomaly and risk detection has an application in almost every organisation, especially with more and more increasing regulatory compliance legislations.
Although the conference had a relatively small audience, the presentations were of a high calibre and the discussions were very stimulating.
