The success of every decision is closely related to the quality of the information that was used to make that decision. For this reason, Data Quality is very closely related to Business Intelligence. Data quality checks and active data quality controls should be embedded into the loading and reporting processes. This can ensure the quality of the information provided to decision makers, as well as present them with statistics on the data quality aspects.
For any executive, making the right decision is crucial in any business environment, especially as the market has become much more dynamic and competitive. Business Intelligence can give an organisation the opportunity to be more effective in its decision making, and to take this even further by mining and using that information competitively.
That is, if BI provides quality information. The quality of any decision is only as good as the quality of the information it was based on (plus a few other influences, of course). If decisions are made on the fly without supporting evidence and based purely on external influences, then poor results will surely follow that decision, and it is for this reason that data quality has come under scrutiny in the BI space.
However, data quality should not be considered as a separate entity. It should form part of the overall BI process. BI, after all, aims to offer the right information (and therefore of good quality), at the right time, to the right audience, for the right decisions to be made. Decisions based on poor quality information can damage the company, or lead to other problems that need fixing. All business processes are directly linked to business data and by using the correct data the correct business processes can be followed, or even be improved, which will automatically lead to better and more effective decision making.
Data quality should therefore always be top of mind for any BI professional, and it should be considered as a critical component of the end-to-end BI process.
When selecting an ETL or Data Integration tool, the data quality management facilities of such a tool must be thoroughly scrutinised. Data quality checks must be implemented as part of every step in the ETL process, as well as between each step in the process. At each step the data values must be inspected for completeness, adherence to data integrity constraints and other business rules. Between each two steps – i.e. between the source systems and staging, between staging and the data warehouse, as well as between the data warehouse and any downstream utilization of the information – consistency and auditing checks must be used to validate each layer’s data content in an auditable manner to the content of the previous layer.
Likewise, BI’s reporting solution must also be used to report on the data quality. The outcomes of the data quality steps – whether data was rejected or cleansed on the fly – must be available online through an easy to access interface, to facilitate reporting, auditing, follow-up or to initiate more drastic actions. There are numerous data quality measures that should be reported, such as accuracy, completeness, timeliness and correctness.
When data quality is tightly integrated with BI, the overall framework will improve other aspects related to BI too, such as following policies, adhering to standards, implementing strategy, ensuring privacy and security and complying with regulatory requirements. The overall management of BI is improved and support thereof is made easier.
