Data silos are pervasive in organisations around the world. At many of the companies where I have worked, these data silos have made it difficult to ensure system integration, data governance, and effective reporting. Following an insightful industry piece I recently came across, for my blog article this month, I decided to focus on and discuss some of the approaches that can be used to break down these walls.
As mentioned, a great industry article to read about this topic is Bob Violino’s “Breaking down data silos for digital success” published on CIO. In it, he uses two key phrases – unifying data strategies and knocking down data (and political) walls.
This highlights that breaking down data silos at an organisation is a strategic imperative. Trying to do so by stealth or adopting a bottom-up approach will be impossible. Furthermore, data silos are often the result of the corporate’s political landscape and make-up of the business. While some of these are a result of the legacy of older organisational cultures, modern businesses looking for integrated and consolidated insights must move beyond those ‘limitations’.
In my experience, healthcare is one of the industries that struggles the most with siloed, point-focused, and unintegrated systems. In his article, Violino uses a children’s hospital in the US as an example of how to overcome these data siloes. The case study examines the hospital’s journey to consolidate 120 separate systems into a single, centralised data warehouse with one reporting tool.
The value in data lakehouse architecture
More organisations are embracing data lakehouse architectures, as opposed to conventional dimensional data warehouses. These are used to systematically collect all relevant structured, unstructured, and streaming data, to store, transform, aggregate, and label it as needed. And finally, to optimise the data for reporting.
At the organisation I am currently at, the data lakehouse architecture is enabling us to ingest data from a myriad of systems much faster and with more agility to adapt to changing requirements. So, instead of an instantiated dimensional data warehouse on top of the data lakehouse, we are using dynamic views and the reporting tool’s semantic modelling capabilities.
This results in putting in place more efficient and business-friendly reporting environments. Not only do these return results faster, but also use fewer resources than it would take to design and implement a dimensional data warehouse and the multi-layer ETL processes to populate it. Of course, in our reporting models, we are still using dimensional principles, but we are not physically instantiating them.
The data steering team
Organisations can make additional strides in breaking down silos by putting in place a dedicated data navigation or steering team. Such a team would help the organisation align data across business areas and establish a data governance function. This will empower decision-makers to ensure the trust, privacy, and security of data while also being able to identify the technology and human resources to use to help build an integrated data architecture.
Such an approach would be especially beneficial to a highly siloed organisation. Having key stakeholders from the various silos participate in a central forum, making key data- and priority decisions, will foster a culture of sharing across organisational boundaries. When key data- and governance-related decisions are shared, and when information about data quality and business gains achieved through centralised reporting is shared, it tends to organically break down the siloes. This approach favours treating data as a shared resource that must be managed accordingly.
Data silos can result in inconsistencies and operational inefficiencies for a business, and their dissolution can ensure the consistency and accessibility of reliable data across the organisation. A centralised data team structure can establish a unified data ecosystem. Breaking down these silos can foster a culture of innovation, facilitating coordination and collaboration between different business areas. This will result in better decision-making, efficiency in providing analytics, and faster service to stakeholders.
Next month, I will be digging deeper into the strategic aspects and other tips on how to break down data silos.
