{"id":7892,"date":"2024-01-11T02:19:36","date_gmt":"2024-01-11T10:19:36","guid":{"rendered":"https:\/\/www.springboard.com\/?p=7892"},"modified":"2024-01-11T02:20:46","modified_gmt":"2024-01-11T10:20:46","slug":"data-quality","status":"publish","type":"post","link":"https:\/\/www.springboard.com\/blog\/data-analytics\/data-quality\/","title":{"rendered":"What Is Data Quality and Why Does it Matter?"},"content":{"rendered":"\n

Data is constantly changing: from business addresses and names to contact phone numbers and email addresses. Data that was useful weeks or months ago quickly becomes outdated and new data needs to be incorporated into decision-making.<\/span><\/p>\n\n\n\n

The purpose of data analysis is to remove bias and use historical data to create actionable recommendations and predictions for the future. But this only works if the data is of high quality, to begin with.<\/span><\/p>\n\n\n\n

This continuous maintenance of changing data is what we refer to as \u201cdata quality\u201d management. One definition of data quality is: \u201cthe planning, implementation, and control of activities that apply quality management techniques to data, in order to assure it is fit for consumption and meet the needs of data consumers.\u201d In other words, ensuring that data can serve its intended purpose within an organization. <\/span><\/p>\n\n\n\n

With quintillions of data bytes generated daily, data quality is a top priority in order to stay competitive in an increasingly digital landscape.<\/span><\/p>\n\n\n\n

Why Does Data Quality Matter?<\/span><\/h2>\n\n\n\n

Poor data quality can cost an organization <\/span>$9.7 million annually<\/span><\/a>. As of 2016, it cost the United States $13 trillion per year. Data quality problems result in a <\/span>20% decrease<\/span><\/a> in worker productivity and explain why <\/span>40% of business initiatives<\/span><\/a> fail to achieve set goals. Incorrect data can harm a reputation, misdirect resources, slow down the retrieval of information, and lead to false insights and missed opportunities.<\/span><\/p>\n\n\n\n

For example, if an organization has the incorrect name or mailing address of a prospective client, their marketing materials could go to the wrong recipient. If sales data is attributed to the wrong <\/span>SKU<\/span><\/a> or brand, the company might invest in a product line with less than stellar customer demand.<\/span><\/p>\n\n\n\n

Historically, errors with data reporting have even led to global catastrophes. The Enron scandal of 2001, which resulted from the non-disclosure of billions of dollars of liabilities and led to the energy firm\u2019s bankruptcy, could have been prevented through better ethical auditing that would have detected the fictitious nature of presented data. <\/span>This incident gives a clear overview about how poor data quality can jeopardize the whole data analytics<\/a> process and cause unbearable damage to the businesses.<\/p>\n\n\n\n

Data quality always matters, but there are certain business contexts that require extra special attention paid to data quality. When engaging in a merger and acquisition, companies need to unify disparate data sources under common data standards, processes, strategies, technologies, and cultures. Data quality is also important for any enterprise resource planning or customer relationship management function. <\/span><\/p>\n\n\n\n

How Can You Preserve Data Quality?<\/span><\/h2>\n\n\n\n

One of the primary responsibilities of data analysts is to guarantee data quality. Data problems can be caused by employee or customer data entry mistakes (the most prevalent cause, according to <\/span>The Data Warehousing Institute<\/span><\/a>), system changes, software errors, or erroneous data integration\/migration. <\/span><\/p>\n\n\n\n

The procedure of examining data for accuracy and completeness is called <\/span>data profiling<\/span><\/a>. Data quality assurance involves removing outliers and irregularities so that the data is representative of the larger picture.<\/span><\/p>\n\n\n\n

The first step in data profiling is making sure there are no missing data fields and that information has been inputted correctly.<\/span><\/p>\n\n\n\n

Some of the <\/span>most common issues<\/span><\/a> affecting data quality are inconsistent formatting of dates and numbers, unusual character sets and symbols, duplicate entries, and different languages and measurement units. For example, a date can be written out or represented numerically in a few different formats\u2014dd\/mm\/yy, mm\/dd\/yy, or \u201cday, month, year\u201d\u2014which would prevent a computer system from properly aggregating and synthesizing data related to time. Many organizations use unicode (universal code standards) for data processing, but sometimes foreign characters come through in an unreadable format and must be converted during the data cleansing process. <\/span><\/p>\n\n\n\n

After importing the data and identifying a problem, data analysts can either accept the error if it doesn\u2019t disrupt the interpretation, remove the error, fix the error, or add a default such as \u201cN\/A\u201d or \u201cunknown\u201d in place of the error. <\/span><\/p>\n\n\n\n

When profiling large volumes of data, data analysts<\/a> will need to construct data hierarchies, rules, and term definitions to understand the interrelationships between types of data. Rules can be simple, such as: \u201cCustomer full name must be capitalized and consist only of letters.\u201d Data profiling verifies what percentage of entries meet the rules and that this percentage is above the threshold required by the organization.<\/span><\/p>\n\n\n\n

Another important check is ensuring referential integrity, that all the table relationships are in agreement. <\/span>Techopedia<\/span><\/a> gives us a good example: <\/span><\/p>\n\n\n\n

When a CUSTOMER_MASTER table contains data like name, social security number, address, and birthdate, and an ACCOUNTS_MASTER table bears bank account information like account type, account creation date, account holder, and withdrawal limits, a Customer_ID field serves as the primary key, linking the two tables. Referential integrity means that a change in Customer_ID in the CUSTOMER_MATER table must be reflected in the ACCOUNTS_MASTER table.<\/span><\/p>\n\n\n\n

What Factors Determine Quality of Data?<\/span><\/h2>\n\n\n\n

A Gartner study lists several <\/span>key factors to examine data quality<\/span><\/a>: <\/span><\/p>\n\n\n\n