Data quality a priority for 2024

Share

Despite the hype surrounding generative Artificial Intelligence (GenAI), I am finding in my industry reading that many industry analysts are predicting that data quality (one of my favourite topics) will remain a key priority for this year – especially when it comes to data management and governance.

Read the rest of this entry »

Crystal-ball gazing for 2024

Share

We have approached that time of year when it’s always interesting to delve deeper into what the analysts see in their crystal balls when it comes to the trends and technologies to keep an eye on for the new year. I have reviewed several industry articles as resources around this, and let’s just say that if all these predictions come true, we will be in for quite a ride in 2024!

Below are just some of the ones related to BI and analytics that I found quite interesting and wanted to share with you.

Generative AI:

This is a term that everyone is very familiar with by now. In a recent Forbes piece, Bernard Marr writes that Generative AI is going to make a huge impact by taking care of most of our menial work. This includes ‘obtaining information, scheduling, managing compliance, organising ideas, structuring projects.’ Of course, he acknowledges that challenges remain around ethics and regulation that must still be solved.

In a Gartner review, Ava MacCartney reckons that ‘by 2026, generative AI will significantly alter 70% of the design and development effort for new web applications and mobile apps.’ While certainly plausible, I’d like to see what the figure is for BI and analytics. In the data sourcing and data engineering space, we are still doing a lot of manual labour that could be automated.

Imagine you can just say: “Get me the data from the CRM and the billing systems and integrate them on Customer ID!” and voila, there you have got data from 60 tables integrated and ready for analysis and to develop models on. “Now tell me which customers are about to churn and recommend a campaign that will entice them to stay.” Ah, we can dream. Gartner places Generative AI, together with Platform Engineering, AI-Augmented Development, Industry Cloud Platforms, Intelligent Applications, and Sustainable Technology under a banner called ‘Rise of the builders’. McCartney believes these technologies will boost the creativity of the communities involved in this type of work.

Developer experience (DevX)

In a sister Gartner paper, Lori Perry writes that ‘the suite of technologies under this theme focuses on attracting and retaining top engineering talent by supporting interactions between developers and the tools, platforms, processes, and people they work with.’ I am all for technologies that will make our data engineers’ work more pleasant. But while powerful, I wouldn’t call the user experience of data pipeline technologies enjoyable and highly productive yet. Perry cites the Value Stream Management Platform (VSMP) as an example of DevX technology that seeks to optimise end-to-end product delivery and improve business outcomes.

She also explores technologies like AI-augmented software engineering that can help software engineers create, deliver, and maintain applications. Furthermore, API-centric SaaS services could potentially be used as the primary method to access these technologies. There is also GitOps, which is a closed-loop control system for cloud-native applications, and other internal developer portals that enable self-service discovery that will increasingly come into the spotlight.

I’m looking forward to seeing these technologies in action to increase productivity and reduce human error in data management.

Responsible AI

In a review of the Gartner Data & Analytics Summit held in Sydney at the end of July, responsible AI emerged as a trend to watch. I like this positive spin on AI and Machine Learning as it covers many aspects of making positive business decisions and ethical choices when adopting AI. These include adding to business and societal value, reducing risk, and increasing trust, transparency, and accountability. Unfortunately, there are way too many case studies where AI and ML models have come up with ethically unsavoury or unusable insights.

Gartner predicts the concentration of pre-trained AI models among 1% of AI vendors by 2025 will make responsible AI a societal concern. The firm further recommends that ‘organisations adopt a risk-proportional approach to deliver AI value and take caution when applying solutions and models. Seek assurances from vendors to ensure they are managing their risk and compliance obligations, protecting organisations from potential financial loss, legal action and reputational damage.’

Data-centric AI

Another interesting topic in the same review is data-centric AI. This is more data-focussed than AI which is mostly based on models, algorithms, and code. Garner refers to data managed specifically for AI solutions. These include data synthesis and data labelling which are employed to solve data-related challenges, such as accessibility, volume, privacy, security, complexity, and scope. In my mind, this is not necessarily new technology, but rather a realisation by AI practitioners that there are aspects related to data governance that are equally important in the AI and ML fields.

What will be interesting is to see how the technology and practices are being adapted to function efficiently and effectively in more fast-moving and fast-changing environments. These environments are reliant on working with large volumes of data, and even data that was not sourced from within the organisation. There are some useful data governance and cataloguing platforms out there, but the challenge has always been to make them work productively at scale. I think it will be crucial for data governance systems to apply AI and ML themselves to function more effectively.

Of course, many technologies and trends also focus on privacy and security. While important, they have not been the focus of this post as I wanted to explore data-specific trends.

The uniqueness of modern data quality management

Share

Last month I addressed how data quality is perceived by different specialists inside the organisation. This month, I turn the spotlight onto what makes modern data quality management different from traditional approaches. Edwin Walker’s Data Science Central article ‘Difference between modern and traditional data quality’ provides an excellent starting point.

Read the rest of this entry »

Data quality is in the eye of the beholder

Share

The quality of the data we work with has a significant impact on the quality of the insights we can extrapolate for the business. Following on from my recent mini-series on the evolving data-related roles, it was interesting to come across Edwin Walker’s article on Data Science Central titled: ‘How do different personas in an organisation see data quality?’ Walker has also written on the topic of ‘modern data quality’ in another article available on the same site.

Read the rest of this entry »

The evolving role of the data scientist

Share

This month, I am wrapping up my three-part series on evolving data-related roles by focusing on one of the fastest changing and popular roles under discussion in the industry today – the data scientist.

Read the rest of this entry »

More evolving data-related roles

Share

Last month, I discussed the evolving roles of the Chief Data Officer (CDO), Chief Analytics Officer (CAO), and the Data Engineer. In my blog post this month, I will delve deeper into several of the other roles related to data engineering, as well as the evolution of the role of the Business Analyst.

Read the rest of this entry »

Examining the continually evolving data-related roles

Share

In my quest to continuously learn more about data-related roles, I have recently come across two very interesting articles that suggest that data-related roles will be changing throughout the remainder of the year. These pieces share some great insight, and I couldn’t resist the opportunity to share my additional views and so in this, the first of a two-part blog series, I will examine the Chief Data Officer (CDO) role, the Chief Analytics Officer role (CAO), and the evolving roles of data engineers.

Read the rest of this entry »

Complexities behind establishing the CDO role

Share

Regular readers of my posts will know that I’m a big advocate of the Chief Data Officer (CDO) role. Also sometimes referred to as the Chief Data and Analytics Officer (CDAO), establishing this role and getting it woven into the organisational fabric doesn’t happen without challenges.

Read the rest of this entry »

Understanding the causes and risks of data waste

Share

In recent years, there has been a significant focus on individuals and organisations to reduce waste in support of global climate change and environmentally conscious efforts. In my blog post this month, I turn the spotlight on a different sort of waste. This is something which every business around the world struggles with – data waste.

Read the rest of this entry »

Older posts «

hope howell has twice the fun. Learn More Here anybunny videos