The increased use of analytics, coupled with an exponential growth in the volume and complexity of data, has changed the business intelligence landscape forever. Business users’ sophistication has extended beyond descriptive reports and dashboards into data exploration, predictive modelling and aspects of cognitive computing. Traditional requirements-driven ETL, data warehouse and reporting approaches do not meet the modern organisations’ insights requirements. In fact, the term “business intelligence” is fast losing popularity. The time has come to seriously consider modern alternatives.
Nowadays, as a recent Gartner report states: “Every business is an analytics business, every business process is an analytics process, and every business user is an analytics user”. But of course, to make analytics useful, the insights have to be made available in a useful and convenient format at the point of action or interaction. This radically changes how analytics is served to and consumed by business users.
Conventional dimensional data warehouses are very good for productionised reports, dashboards and structured drill-down along the defined dimensions. However, despite the promises of conformed dimensions that integrate data across functional boundaries, dimensional data warehouses still resulted in data silos – not because of shortcomings in the modelling approach, but because of technology limitations, lack of foresight in what may be required down the line and especially the laborious modelling and coding required. The time and effort of ETL implementation seriously limits the flexibility to adapt to fast-changing business requirements. Let’s face it, traditional data warehouse platforms and processes were simply not designed for the current pace and dynamic nature of business change, nor the exponential growth in terms of the sources, volume and complexity of data. But five years ago there were no appealing alternatives.
However, those limitations have been erased by the affordable availability of large in-memory and specialised data stores, together with toolsets with very powerful parallel processing capabilities. Some of these are called modern analytics platforms. (Note that some of these commercially available platforms have wider use cases than analytics, but that’s another story for another blog post.)
Definition
So what is a modern analytics platform? It is an IT-enabled self-contained architecture that enables non-technical users to manage full-spectrum analytic workflows themselves. I discuss analytic workflows in detail below.
In addition to the characteristics expected from a conventional BI ecosystem, such as performance, availability, security, accessibility, etc., a modern robust analytics platform has additional characteristics:
- Self-service: Business users can identify, obtain, query and analyse data in order to get to their own analytical insights. The key differentiator between this and self-service reporting lies in users being able to identify and obtain data themselves.
- Flexibility and extensibility: Users can easily add new data sources and new analytical insights.
- Guidance: Built-in intelligence intuitively assists users to get the data they need and to develop the reports, visualisations and analytical insights they need to improve the business.
- Trust and transparency: Data is well documented and well governed to the extent that the users can trust that it is legitimate and correct. This includes data quality, data lineage and business meaning.
- Connectivity: Access to all types of data, in databases, external data sources, social media streams, logs, unstructured files and more.
- Collaboration: Users can share insights through a contemporary system of insight.
- Mobility: Users can access data and insights from a wide variety of platforms and mobile devices, with a consistent user experience across these.
Analytical workflow
So how do you use modern analytics platform? A user typically works through five steps of an analytic workflow, namely data sourcing, data ingestion and preparation, content authoring, analytics and insight delivery. The following is an illustration of an analytical workflow:
Data sourcing
In addition to a large array of structured data, both from on premise applications and offshore data sources, including relational and NoSQL databases, the analytics platform provides access to alternate data sources such as:
- Un-modelled data in Hadoop file systems, whether on premise or in the cloud. This can contain replicated transactional data, syndicated data, freely available data, sensor or machine-generated data and organisational unstructured data traditionally considered beyond the scope of BI technologies (such as emails, collaborations and documents).
- Cloud applications.
- Social media streams.
- JSON/XML data.
- Data in Microsoft Excel and comma-separated values (CSV) files.
Data Ingestion and Preparation
A modern analytics platform does not require users to only access pre-modelled and pre-stored data through a predefined semantic layer. In fact, it allows skilled users to iteratively explore, ingest and prepare data from any source, with speed and agility. This is done through powerful data discovery and on-the-fly data access and data integration capabilities. The latter is typically performed through a federated query processing component that breaks a logical user query into physical sub-queries, requests and controls their autonomous executions on various remote data stores and then integrates the results before presenting it as a logical result back to the user, all within acceptable response times.
As part of ingestion, conventional ETL jobs may be kicked off to run seamlessly in the background, to transform and move data into an appropriate store within the analytics platform. This is typically done if continuous and real-time access to the source data is not desired or allowed for reasons like interference (e.g. locking), performance, accessibility, system isolation or security. However, the ETL jobs are not coded by the user – they are defined and configured (typically using a metadata facility) and then run automatically as and when required.
Users can therefore prototype their own data models and test new data sources in combination with existing data structures without first requiring the BI team to design dimensional models and design, code, test and implement cumbersome ETL processes.
As part of the ingestion, the users may apply “smart data preparation” steps for analytics, including:
- Agile or on-the-fly modelling, transformation, data cleansing and data enrichment.
- Inference processing, such as custom grouping and calculations, defining relationships between dataset entities (i.e. applying schema-on-read), classification of dimensions or measures, applying geocoding, converting dates and times and data profiling with data quality improvement.
- Data blending, which entails joining datasets with the purpose of creating a federated or logical view of the data, for example using manual definitions of joins or allowing the facility to detect and infer sometimes undocumented joins and relationships.
Of course, an existing semantic layer or dimensional data warehouse can also serve as a data source in a modern analytics platform, where users can then integrate objects defined in the semantic layer with other data sources using the data preparation capabilities outlined above. So a past investment in data warehousing and BI platform is not lost at all – it is absorbed in and enriched through the modern analytics platform.
Content Authoring
New authoring capabilities allow users to perform the following, without assistance from the traditional BI team:
- Develop interactive analytic content that intuitively makes sense.
- Design and create interactive analytic reports and dashboards.
- Create storyboards or narratives, often using data visualisation paradigms.
Whereas traditional BI platforms primarily address descriptive reporting requirements, modern analytics platforms enable nontechnical content authors to find important insights using a broad range of more advanced analytical capabilities. You do this by means of intuitive search, drag-and-drop authoring, interactive exploration and visualisation and embedded advanced analytics. Much of the analytic content that will be utilised by a growing population of business users will originate as interactive explorations.
Analysis
The types of analysis that a modern analytics platform supports, include:
- Iterative modelling of data through visual exploration and visualisation.
- Navigation of analytic dashboards and embedded analytic content.
- Interactive consumption and exploration of storyboards and infographics.
- Dynamic analytic narratives describing key insights.
Business users and analysts are also becoming more receptive to new ways of interacting with data through automated recommendations and data discovery, new visualisations types and natural-language Q&A.
Insight Delivery
Without a modern analytics platform, interactivity is limited and often restricted by the content author, so users frequently export data to other formats (such as Microsoft Excel) to continue to analyse it and to combine it with other sources.
Enabling sharing, collaborating and socializing content are some of the key attractions of modern analytics platforms. These are convenient for users to reach consensus about the interpretation of results and for deciding the best resulting actions. This includes live chats, annotations, conversation timelines, likes/dislikes, content rating, ability to follow, recommendations and so on.
You can enhance the collaboration by stringing together insights as interactive and shared storyboards that “walk” the other collaborators through a series of insights that lead to a conclusion and recommended actions. The platform allows you to define triggers and thresholds for automatic alerts and notifications when certain user-specified conditions are met or exceeded.
Natural-language generation (NLG) is emerging in modern platforms – it automatically generates context-sensitive narrative that focuses the user on what is statistically significant in the data. This increases the scalability and reach of analytics within the organisation. Visualisations with easy-to-understand narratives accelerate the time to insight. They also ensure that the intended conclusions are made by users who may have limited analytics skills.
Benefits
Modern analytics platforms enable fast time-to-insight, the ability to change and evolve at the speed of business, and the ability to use information as a strategic competitive advantage.
The innovation that comes with an agile and iterative modern analytics platform delivers the necessary building blocks to increase analytic maturity and derive value from data. Organisations can gain new, previously undiscovered insights with which to improve decision making. It also creates opportunities for differentiation and new revenue streams as new insights are packaged and extended to customers, partners and suppliers.
BI leaders that fail to adopt modern analytics platforms will be unable to respond to the faster pace of business and to users’ voracious appetite for fast insights from more data sources than ever before. As we said before, a traditional centralised and rigidly controlled data warehouse and BI capability cannot meet these new, time-sensitive requirements.
Risks
However, despite the obvious benefits of moving to a modern analytics platform, you must assess the potential risks as well:
- The need for significant change management.
- Even though the users are more self-sufficient you may still need additional IT resources to support the more complex environment, especially with regards to connectivity and accessing different data sources.
- Integrating existing BI infrastructure with a new modern analytics platform can be complex — especially when it involves multiple locations and business units, and different information latency and data quality requirements.
- The risk of information chaos resulting from inadequate information management (data governance, data consistency, data quality and so on), insufficient expertise and the initial low maturity levels of the platform users.
Concluding remarks
It is important to note is that simply adding modern capabilities, such as visualisation or collaboration, to a conventional BI platform that still requires traditional data access and ETL-driven data transformation and loading does not make that platform modern. All aspects of the analytic workflow must be modern for an analytics platform to be categorised as modern.
We have reached the point where the demand for modern analytics platforms, driven by the requirement to make analytics pervasive without causing chaos, has become mainstream across all industries, business domains and company sizes.
I will cover the technical components of a modern analytics platform in a subsequent blog post.
References
Technology Insight for Modern Business Intelligence and Analytics Platforms, by Gartner analysts Josh Parenteau, Rita L. Sallam, Cindi Howson, Joao Tapadinhas, Thomas W. Oestreich, Kurt Schlegel, published 23 October 2015.
Monetize your Data with Business Intelligence, an IBM / Ziff Davis white paper, available at http://public.dhe.ibm.com/common/ssi/ecm/yt/en/ytl03392usen/YTL03392USEN.PDF
