{"id":3957,"date":"2018-08-08T12:49:07","date_gmt":"2018-08-08T19:49:07","guid":{"rendered":"https:\/\/www.springboard.com\/?p=3957"},"modified":"2025-04-23T01:29:01","modified_gmt":"2025-04-23T08:29:01","slug":"data-quality-management-tips","status":"publish","type":"post","link":"https:\/\/www.springboard.com\/blog\/data-science\/data-quality-management-tips\/","title":{"rendered":"Data Quality Management: What You Should Know"},"content":{"rendered":"\n

The heuristics we learn in the classroom are often just the tip of the iceberg. Sooner or later, if we want to deepen our study, it becomes necessary to bend or even break the rules that got us so far in the first place.<\/span><\/p>\n\n\n\n

This is certainly the case with data quality management. Data quality, defined as \u201ca perception or an assessment of data’s fitness to serve its purpose in a given context,\u201d might <\/span>seem <\/span><\/i>straightforward at first but is actually rather difficult to evaluate and maintain. This is in part because the threshold dividing high-quality data from low-quality data depends on a number of variables, including the data\u2019s intended use (or uses) and context (which may change over time), as well as a subjective understanding of the data\u2019s accuracy, completeness, and reliability.<\/span><\/p>\n\n\n\n

This is where data quality management (DQM) comes in.<\/p>\n\n\n\n

What Is Data Quality Management?<\/h2>\n\n\n\n

If you’ve ever heard a variation of the phrase “a system is only as good as the data it contains” it’s likely the speaker was talking about data quality management.  At its core, DQM is simply a set of protocols and processes that help a business collect, clean, analyze, store, and distribute data as consistently as possible.<\/p>\n\n\n\n

While the clean, uncomplicated datasets we encounter in examples<\/a> and courses<\/a> may help us grasp data quality fundamentals, they are unlikely to prepare us for the more nuanced scenarios we find in the workplace. Let\u2019s look at how basic quality control practices mature in these more complex data environments.<\/span><\/p>\n\n\n\n

Stage 1: Referential Integrity<\/span><\/h2>\n\n\n\n

When you first start working with relational databases, you\u2019ll come across something called a referential integrity check. It\u2019s a data quality control feature built right into the database that <\/span>enforces the relationships between tables<\/span><\/a>: \u201cany foreign key field must agree with the primary key that is referenced by the foreign key.\u201d If you\u2019re unfamiliar with these terms and missed <\/span>our post on SQL table joining<\/span><\/a>, here\u2019s a classroom-style example to illustrate the concept. <\/span><\/p>\n\n\n\n

Let\u2019s imagine we\u2019re a medical office, and we have patient records stored in a number of database tables, among which are the PATIENT table and the ADDRESS table. Because our PATIENT table references the ADDRESS table, linking each patient to his or her address, our database will require that we supply an address key in order to add a new patient to the system.<\/span><\/p>\n\n\n\n

Given the address table below, the database would bar us from adding the highlighted patient record because its address id does not correspond to an address id in the ADDRESS table.<\/span><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"integrity\"<\/figure>\n\n\n\n

The referential integrity constraint is predicated on the idea that quality data <\/span>exists where it is supposed to exist<\/span><\/i>. Because address #22222 isn\u2019t where it\u2019s supposed to be (in the ADDRESS table), the integrity of the entire Jess Keating record is questioned.<\/span><\/p>\n\n\n\n

But what if our fictitious medical office is actually an emergency room at a hospital? Perhaps Jess Keating came in on a gurney in critical condition, and although we know her name from the student ID card on her person, we don\u2019t know her address and cannot ask her for it. We need to be able to add her to the system without an address.<\/span><\/p>\n\n\n\n

This might seem like an extreme case, but unknowns pop up all the time in the real world. Jess could just as easily have shown up as an outpatient and simply neglected to give her address before being called in to see the doctor. Or perhaps she was giving her address over the phone but got cut off. <\/span><\/p>\n\n\n\n

Why not just leave her address blank or NULL, then? The absence of a value isn’t the same thing as a conflicting value, is it?<\/p>\n\n\n\n

Well, this depends on context, and for the purposes of our example, a non-existent address is a value. Moreover, it’s an incorrect value, as Keating does live somewhere.<\/p>\n\n\n\n

It is not in our best interest to populate tables with false information, and yet the practice is pretty common in the real world<\/a>. This is a simplistic example compared to the kinds of unknowns that routinely surface in high-volatility industries like finance and insurance, where the repercussions of entering a false empty or NULL value can be costly indeed. In these cases, insufficient data is preferable to the alternative\u2014even though it lacks completeness\u2014because it is more accurate and therefore more appropriate to the business context. We have progressed from a referential integrity issue to a data integrity issue.<\/p>\n\n\n

Get To Know Other\tData Science Students<\/h4>
\"Abby<\/a>

Abby Morgan<\/p>

Data Scientist at NPD Group<\/p><\/div>

<\/div>

Read Story<\/a><\/p><\/div><\/div>

\"Meghan<\/a>

Meghan Thomason<\/p>

Data Scientist at Spin<\/p><\/div>

Read Story<\/a><\/p><\/div><\/div>

\"Jonathan<\/a>

Jonathan King<\/p>

Sr. Healthcare Analyst at IBM<\/p><\/div>

Read Story<\/a><\/p><\/div><\/div><\/div><\/div>\n\n\n\n

Stage 2: Data Integrity<\/span><\/h2>\n\n\n\n

Where in Stage 1 we were most concerned with the integrity of our table references, we are now going to loosen those requirements in order to prioritize <\/span>data integrity<\/span><\/a>\u2014that is, the internal consistency and lack of corruption in the data itself. <\/span><\/p>\n\n\n\n

Let\u2019s refer back to our hospital example for a moment. If Jess Keating is currently undergoing a life-saving procedure in our emergency room, it is important that we have a record of her, <\/span>even<\/span><\/i> if that record is incomplete. Failure to record her visit as accurately as possible could have catastrophic consequences, so in order to bypass our database\u2019s referential integrity check, we need to implement an override.<\/span><\/p>\n\n\n\n

In our case, the override would take the form of a stub record, a temporary stand-in for the real address we don\u2019t yet have.<\/span><\/p>\n\n\n\n

\"data<\/figure>\n\n\n\n

This dummy record satisfies the database\u2019s referential integrity requirements while simultaneously accounting for the fact that we have a patient by the name of Jess Keating with no known address. <\/span><\/p>\n\n\n\n

But now we have another data quality problem on our hands. Jess Keating\u2019s address, by being blank, is effectively incorrect. Ordinarily, <\/span>no <\/span><\/i>data entry would be preferable to <\/span>false <\/span><\/i>data entry, but in this case, circumstances and the referential integrity constraint force our hands. The stub record must eventually be corrected, not only to restore the record\u2019s integrity but also to keep from triggering a series of patient processing issues. The hospital needs to instate a system that will ensure that the record gets filled. We now progress from data integrity to data quality.<\/span><\/p>\n\n\n\n

Stage 3: Data Quality<\/span><\/h2>\n\n\n\n

So far, we\u2019ve gone from determining that our tables are accurate to verifying that our data is accurate. But to assess all the factors<\/a> contributing to data quality\u2014accuracy, completeness, validity, and relevance, among them\u2014we must actually understand what the information is <\/span>for<\/span><\/i>. This, in the data world, is referred to as \u201csemantics.\u201d Whoever designs the processes that will ensure Jess Keating\u2019s address is collected and added to the database will have to understand who needs that information and when and why. We are all familiar with the utility of an address, but not all data points are as transparent to the layperson. Data management teams need to understand what the data means and how it’s being used in order to establish data quality metrics and safeguard its quality.<\/span><\/p>\n\n\n\n

In our example, the data quality check might be that the receptionist at the ER\u2019s discharge desk is required to fill in any vacant fields before the patient is released. The hospital <\/span>could<\/span><\/i> rely on receptionists to scan all the necessary paperwork and identify where information is missing, but this leaves a lot of room for human error. An application programmed to perform this scan on the receptionist\u2019s behalf is better equipped to catch all the missing information, and it is this program that a data quality analyst might design based on the hospital\u2019s processing needs.<\/span><\/p>\n\n\n\n

The need for automated quality check compounds when we introduce more complex data quality concerns, such as duplicate records. Let\u2019s say this isn\u2019t Jess Keating\u2019s first visit to the hospital; she\u2019s already in the system under <\/span>Jessica <\/span><\/i>Keating. We don\u2019t realize this at first because all we have is her student ID, which identifies her as Jess. When Jess is discharged from the ER and we collect her address, it\u2019s possible we\u2019ll get something different than what we have on file for Jessica; she may have moved recently. <\/span><\/p>\n\n\n\n

In this scenario, the patient\u2019s social security number would lead us to consolidate the duplicate records, but not all real-world scenarios will have such a convenient unique identifier! In less certain cases, data stewards may need to utilize data quality measures and run a program checking for patterns in multiple fields (e.g., records with the same address, same last name, and same first letter of first name), flagging the potential duplicates for investigation. <\/span><\/p>\n\n\n\n

Data quality management is a key concept in the field of data science<\/a>. Subject matter experts and data specialists<\/a> or data scientists<\/a> need to work <\/span>together <\/span><\/i>to ensure data quality at the highest level. Referential integrity and data integrity are merely stepping stones on the path to a top-quality data system that can accommodate real-world information requirements.<\/span><\/p>\n\n\n\n

Nicole Hitner is a content strategist at Exago, Inc., producer of embedded business intelligence for software companies<\/a>. She manages the company\u2019s content marketing, writes for their blog<\/a>, and assists the product design team in continuing to enhance Exago BI.<\/em><\/p>\n\n\n\n

Since you\u2019re here\u2026
<\/strong>Curious about a career in data science? Experiment with our free data science learning path<\/a>, or join our Data Science Bootcamp<\/a>, where you\u2019ll get your tuition back if you don’t land a job after graduating. We\u2019re confident because our courses work \u2013 check out our student success stories<\/a> to get inspired.<\/p>\n","protected":false},"excerpt":{"rendered":"

The heuristics we learn in the classroom are often just the tip of the iceberg. Sooner or later, if we want to deepen our study, it becomes necessary to bend or even break the rules that got us so far in the first place. This is certainly the case with data quality management. Data quality, […]<\/p>\n","protected":false},"author":45,"featured_media":3986,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_eb_attr":"","_eb_data_table":"","footnotes":""},"categories":[67],"tags":[],"marketing_tags":[],"class_list":{"0":"post-3957","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-data-science"},"acf":[],"_links":{"self":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts\/3957"}],"collection":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/users\/45"}],"replies":[{"embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/comments?post=3957"}],"version-history":[{"count":4,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts\/3957\/revisions"}],"predecessor-version":[{"id":56567,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts\/3957\/revisions\/56567"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/media\/3986"}],"wp:attachment":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/media?parent=3957"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/categories?post=3957"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/tags?post=3957"},{"taxonomy":"marketing_tags","embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/marketing_tags?post=3957"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}