Useful Thinking About Big Data

Ever wonder why “Big Data” is referred to in terms of somebody’s solution, tool, application, utility?

And why that doesn’t offer much value to you?

You know there must be some value...otherwise why all the noise?

The best way to define Big Data is in terms of how you are currently using it. You aren’t? Bugger!

In that case, “Big Data,” like the blind guy said, “is like a wall.”

Like the blind guys and the elephant, we are going to have to construct a model that is useful and entertaining for you. Here goes:

Web 1.0,   Web 2.0,   Web 3.0

Web 1.0 was that gray background on your computer screen with black type, “Hi, I’m computer programmer. Here is a picture of me and my dog!” That was 1995. I was artsy, so I had flaming letters, proving I had more research than taste.

Today, Web 1.0 continues on corporate websites with tabs like, “Our Team,” “Vision,” and “Shopping Cart.” A billboard in cyberspace, hiding the countryside above the big tubes

.

Web 2,0 is adding two-way conversations, the back and forth, often with hilarious or cringey  results. From trolling right up to creating new ideas, organizations, and things.

The Cluetrain Manifesto alerted us to the coming of Web 2.0, with the first thesis, All markets are conversations. If you've staked your claim on Google Plus or LinkedIn, you are a Web 2 resident.

Web 3.0 is repurposing what has already been loaded into the web.

For Web 3.0, the words, pictures, and ideas have often already been loaded into the Internet. Not all, but the ease of using existing data makes gathering whatever else is needed much lower cost. Repurposing existing data creates another round of value.

Examples are pictures of winsome cats and a caption, “I can has cheezburger?” to all the privileged and then stolen boilerplate in the legal universe, to everything everyone knows about our health, wealth, and happiness. That whole pile is big data, and the people telling you about it have clever hacks to connect pieces of that data into useful piles.

Scary? Could be.

Miraculous? Sometimes.

Will people be swindled? Always.

Will people be helped? More.

But the work of making it work is not magic. It is figuring out what goes where.

Doc Searls, an author of the previously mentioned Cluetrain, has been working on Vendor Relationship Management (VRM).

Customer Relationship Management allows siding salesmen to know what time you sit down to dinner so they can call. VRM allows you to make the best choice when you want something, and not spend time with it when you don’t.

Thankfully, Big Data and VRM are largely an open source collaboration, so you can find it and see what others are doing.

And yes, it’s bad what fast buck artists, lawyers, and legislators are trying to do to you. But yes, it’s good how you’ll be able to improve your life with the resources that are available to you.

The internet is so big it creates an opportunity for advantage from scaling and availability that we have never seen before. Sturgeon’s Law (Ninety per cent of everything is crud) notwithstanding, there are tremendous opportunities.

What we learned from the App Economy is:

1. First in gets the biggest reward,
2. Nothing is ever so completely finished that it can't be radically improved, and
3. Collaboration outproduces closed systems.

Be on the lookout for a use of Big Data/Web 3.0 that will help you a lot.

Open Source Leadership – A useful way of seeing the world

Join us on Monday July 15^th for The Changing Social Paradigm - How It Affects the Job Market, for details about this free presentation: 40Plus Washington, DC.

Don't File It – Use That Data

Big Data is a means to extract useful information from great volumes of data collected from various inputs. If the data is available and accurate, the distilled information can be useful if the project is well conceived.

Last week we attended a Big Data session put on by NCA-GTUG to highlight some successful in-use projects with Splunk boiling down huge quantities of continuous real-time data into useful information about monitored activities – almost real-time. Fascinating.

Next day I participated in a GovLoop Webinar on Optimizing Service With Big Data, which outlined the expanding use of the Big Data approach in the Government sector – and an example of effective use of police data to reduce crime.

The city of Santa Cruz applied Big Data tools to 10 years of collected police crime data. They cut the data by type of crime; they sliced it by location; they diced it by time, day of week, day of month, and month of year, to get a detailed picture of the characteristics of crime occurring in Santa Cruz.

Then the process was reversed to create when, where, and what predictors of crime probabilities as a data based tool to apprehend or prevent area crime.

By manipulation of large volumes of existing data, the police department created a knowledgebase analogous to the instincts of long-service patrolmen with a 'feeling' about crime activity and targets. This predictive tool is intended to help the SCPD be more effective in the reduction of crime. Of course, personal experience is what sets the Doers apart from others – this data approach is a supplement, not a replacement for experience.

The results bear out this supposition: property-based crime is down by 12% and burglaries down by 25% since implementing the project. Good results, which supplement the other tools available to the police in doing their jobs – like when the two-way radios replaced the call boxes on the street.

This example demonstrates a valuable subsequent use of gathered data to glean information about patterns, trends, or other relationships among the measured items. Two important element to a successful project with massive volume of data is the relevancy of the project to get usable information and the completeness and accuracy of the input to get sound information.

What are some Big Data applications that could create valuable information for your operations or future planning?