Sunday, April 24, 2011

Data Cleansing every quarter?

@jschwa1 Data cleansing every 3 months? http://ow.ly/1i0vd - Someones not addressing the right problem!

This is a clip from a recent tweet from Julian Schwarzenbach of Data and Process Advantage Limited (DPA)

My response to his tweet was " I can see validity [of quarterly cleansing] esp. if the data is from external sources like customers". 

I can see where Julian and others might see quarterly cleansing as a lack of attention to the main issue.  His assertion is that if you need to cleanse your data every three months, maybe you have other issues you could address "up-stream" that would alleviate the need to perform cleansing so often.  I want to say that I completely agree with this especially when the data is created, maintained and distribute within an organization. 

However, there are quite a few occasions when data is not created or even maintained "in-house" and in this situation it is a good practice to cleanse this data at practical intervals.

An example of data created from outside the organization is customer data.  Frequently this data is entered directly by the customer into a database from web-enabled order entry and customer service forms. 

Julian would interject to say increase data quality validation on the web forms and alleviate the need for quarterly cleansing.  Agreed!  However, this doesn't assure high quality data and there is a prevailing thought in the industry that user interface validation decrease the "customer experience" so there is a reluctance to implementing these types of validation.

When considering the statement above in this context, I can see where quarterly data cleansing efforts can be a feasible, practical and even wise practice.

1 comment:

  1. [...] #dqchronicle: Data Cleansing every quarter? – @jschwa1 Data cleansing every 3… http://thedataqualitychronicle.org/… #dataquality [...]

    ReplyDelete

What data quality is (and what it is not)

Like the radar system pictured above, data quality is a sentinel; a detection system put in place to warn of threats to valuable assets. ...