Big data, are we ready?

Big data, little data, we live in a world that is saturated in data. These little pieces of electronic flotsam trail us like the wake of a boat. We carry our cell phone and it records where we are. We open an email and immediately that fact is recorded and compared to the mailing list, as are any of our clicks (or taps in the mobile age) made on the links. We visit a web page, or see an ad, or sign up for any number of services. Some of this data is explicit (like signing up for a newsletter service) and some of it is behavioral and simply the side effect of existing online (like ads recording that we saw them and choosing which should be the next ad to which we are exposed).

A recent report from the PEW Internet project tries to measure the differing opinions from some of the thought leaders on big data to see what the future of this technology might mean for mankind. They divided the results into those that predicted that big data would improve our lives and those that predicted that it would degrade them.

Points from the Positive Side

Some of the benefits they believe we will reap:

• By 2020, the use of Big Data will improve our understanding of ourselves and the world.
• “Nowcasting,” real-time data analysis, and pattern recognition will surely get better.
• The good of Big Data will outweigh the bad. User innovation could lead the way, with “do-it-yourself analytics.”

Some say the limitations of Big Data must be recognized and they warn:

• Open access to tools and data “transparency” are necessary for people to provide information checks and balances. Are they enough?
• The Internet of Things will diffuse intelligence, but lots of technical hurdles must be overcome.
• In the end, humans just won’t be able to keep up
• Humans, rather than machines, will still be the most capable of extracting insight and making judgments using Big Data. Statistics can still lie.

Points from the Negative Side

Some of the ways that big data will harm our lives and societies:

• Take off the rose-colored glasses: Big Data has the potential for significant negative impacts that may be impossible to avoid. “How to Lie with the Internet of Things” will be a best-seller.
• We won’t have the human or technological capacity to analyze Big Data accurately and efficiently by 2020.
• Respondents are concerned about the motives of governments and corporations, the entities that have the most data and the incentive to analyze it. Manipulation and surveillance are at the heart of their Big Data agendas.
• The rich will profit from Big Data and the poor will not.
• Purposeful education about Big Data might include priming for the anticipation of manipulation. Maybe trust features can be built in.

But no group is completely cohesive, from this group we also hear:

• Humans seem to think they know more than they actually know. Still, despite all of our flaws, this new way of looking at the big picture could help.
• So the best-intentioned of humans will try to use Big Data to solve Big Problems, but are unlikely to do well at it.
• Computer science, data-mining, and a growing network of sensors and information-collection software programs are giving rise to a phenomenal occurrence, the knowable future.

So, what do you think? Read the PEW study and come to your own conclusions. If you have time, come back here and post your thoughts.