Data Science – Organisational Impact

A lot has been written about the skills and tasks of the data scientist, but what is also very important to consider is the impact of the role on the organisation, and vice versa.

New user-friendly and intuitive tools make it possible for a larger part of the workforce to take part in the data discovery and data analysis processes. Some trends are easy to point out – you can just plot the data. But as Hal Varian pointed out in a milestone interview in 2009, for the richer and more elaborate stories you have to dig much deeper into the data. This is where “real” data scientists play a role.

But what are the cultural implications for an organisation to embrace data science?

Data science is most effective when data-informed decisions are a top priority.  The decision-makers must want to see the supporting data when making strategic decisions. Collaboration must be driven from the decision-makers and the decision-enablers alike, and the organisational culture must allow and encourage these groups to bridge the gap. The business must also be able to timeously and effectively act on the generated insights.

Data scientists should be encouraged to become key power-players in the organisation. On one hand they require system level access to detailed data, while on the other hand they must inform executives of trends and other discoveries that affect the organisation’s strategies. This challenges traditional hierarchies, politics and boundaries.

Data scientists are very likely to be involved in various stages throughout the data life cycle. In order to get to the insights, and deliver them instantaneously to key decision-makers, they do not have the luxury to wait for the “normal” data-related bureaucracies. They may challenge data policies and rules which for years have been accepted as gospel. They may insist on access to data outside the organisation. These “deviations from the norm” will cause stress if the organisation is not culturally ready, willing and able.

Prolific interaction with data scientists, together with the availability of more user-friendly data exploration tools, will naturally make self-service BI more prevalent in the organisation. As a consequence, IT will need to support bigger data clusters.

Part of the role of the data scientist is to promote a data-driven culture, by exposing the data, making it relevant to the appropriate people and showing them what can be done with it. A rather large audience need to become aware of the approach and the results in order to advance the culture. The data scientists can use collaboration tools like bulletin boards, micro-blogs, activity streams and other internal social media to broadcast the message and improve relations with other departments. However, some organisations disallow the use of such facilities.

Where the data science role is located in the organisation can also affect its effectiveness. They may play a key role in product development (e.g. in organisations such as LinkedIn, Amazon and Google, where the products themselves are data-driven and data-dependent), but locking them inside the product organisation my limit their role across functional boundaries. In some organisations the Centre of Excellence (COE) approach works well, but in others it leaves gaps which are later very hard to overcome. The optimal placement will depend on the company’s culture and structure, but great care must be taken to avoid the potential political pitfalls.

Data scientists seem to require a much wider degree of partnership throughout the organisation than traditional BI or IT staff members, as they have to collaborate across functional silos to be more effective.

Data scientists are still relatively easy to manage using traditional project approaches in areas like product development, but when it comes to data discovery – the quest to find interesting stories in the data – the organisation needs to be a lot more fluid. Data discovery cannot be easily tied to strict project and delivery schedules as it requires time and freedom to explore, hypothesise, test, validate and present findings. Even the time to locate and make sense of the underlying data may be hard to predetermine.

However, you also need scientific rigor. In another keystone blog Nathan Yau posted that data science is not like “throwing sticky spaghetti against a wall and seeing what happens”. Experimentation in the data science world refers to hypothesis testing, investigative analysis and semi-structured discovery, rather than random browsing and digging. So data scientists need to be managed with the right balance between experimental freedom and scientific method.

The lack of access to organisational data is one of the biggest techno-social barriers to data science. In a recent EMC survey, BI professionals and data scientists alike felt that very few organisations allow employees to run experiments on data, thereby undermining the company’s ability to rapidly test and validate ideas.

Organisations have to realize that the investigative, creative and communication aspects of data science cannot be automated. If you cannot even automate advanced analytics prediction models, how can you possibly automate investigative analysis?

The organisation has to be receptive and accommodating to the cultural change required to manage, make use and profit from data science’s potentially rich rewards.