The Data Explosion: Opportunities and Challenges Abound

It is an interesting time to be in data management.  There are more sources of data in so many varied formats than ever before.  There are new tools continuously evolving at light speed.  There is the promise of opportunity and with it enormous challenges.

With regard to the opportunities, one of the most interesting things I see developing is increased access to customers.  From traditional to mobile platforms, there are more avenues to interact with customers, presenting an opportunity for product and service providers new ways to measure their effectiveness.  I have starting researching things like sentiment analysis which is an example of how access to customers and data explosion provides insight into product / service perception.

With regard to the challenges, performing analysis on this data requires tools, methodologies, and resources that are very unique and unconventional.  For most organizations, it will take some time to align the resources to perform meaningful analysis.  That is not even taking into the account the budget that needs to be set aside for this activity.

While the technology industry is thrilled with their new story filled with magical elephants and all the promise of a new reality, the boots-on-the-ground in data management must feel like a deer caught in the headlights of an on coming 18 wheeler at 90 mph.  To some the data explosion must feel like fireworks against the warm summer sky, to others the explosion must feel like the pounding of cannon fire against the office wall.

What I think it is very important to realize is that this data explosion is really both at the same time.  We need to be realistic and remember that while there is a lot of data out there and with it comes the promise of gaining new insights, this presents significant challenges to organizations in just how they are going to roll this into the mix of things that already need to do.

I intend on keeping my eye on what organizations come out winners and, maybe even more so, what organizations come out as losers in this new data frontier.  One of the things I intend on paying particular attention to is ROI.  What it costs to do this well and what it produces.

Until I see what that looks like, I am going to hold off getting giddy about big data / no sql … what about you?  Are you “all in” or waiting to see how this goes?

Technorati Tags: big data,data management,data mining,ROI,customer analysis,CRM,sentiment analysis