Unbelievers in the age of Big Data

Cold hearted orb that rules the night, Removes the colours from our sight, Red is gray and yellow white, But we decide which is right. And which is an illusion?

The Moody Blues wrote those famous lyrics nearly 50 years ago and way before the term ‘Big Data” was coined. These days with all kinds of social sentiment, sensory, satellite and other data which clearly help identify red as red and yellow as yellow, we still seem to want to decide which is right and which is an illusion.

Take what happened in Florida last week. Matt Drudge, the conservative blogger questioned if the government was exaggarating the intensity of Hurricane Matthew. I live on the western coast of Florida and was not affected much by this storm, but having lived through several close calls, I was pleased to see our governor (who may be even more conservative than Matt), sound the alarms loud and clear. Over 1.5 million Floridians faced evacuation orders. But the Governor then fought pleas for extending the voter registration deadline and a court had to intervene.

That was one of the largest Florida evacuations in recent memory.  One of the most impressive achievements of the National Hurricane Center is that its ‘track forecast error” has been steadily dropping over decades. The improvements in track forecasts have meant that hundreds of miles of coastline have not been evacuated and we have saved millions of dollars in emergency services. As I found out when I wrote a case study on the NHC in The New Polymath, it has to collect truly “Big Data” via satellite imagery, flights by the Hurricane Hunters, sensors on buoys in the water, dropsondes parachuted through storm clouds and other sources. It uses supercomputing power to create multiple models of likely tracks ( you see them as spaghetti tracks on your TV). It goes back at the end of each season and audits its forecasts.

And yet, we let our politics question the men and machines at the NHC. I have noticed a bothersome trend with my right leaning friends. They are suspicious of any government sourced data – they are afraid to give President Obama any credit and, in turn, potentially help Hillary Clinton’s chances.

But my liberal friends are no better. They are so convinced of the “middle class squeeze” and want more social programs that they refuse to believe Big Data from the IRS, the Bureau of Labor Statistics and the Census Bureau that I mined for my new book, Silicon Collar. The data shows plenty of opportunity for anybody with some initiative

· if you leave out the top 5%, the rest reported $ 6 trillion in AGI or $ 8 trillion in income to the IRS

· for 3+ years, the BLS has reported at least 4 million unfilled jobs every single month

· this economy has 40+ million jobs in franchises, platforms (Apple, eBay, Uber etc.), new services (alternative health, ethnic grocers etc.) which are not being tracked very well, but providing opportunities for many at $20, 100K, eveh higher a year

Most concerning is many of my tech savvy friends who do not want to accept the century of Big Data of research for the book which shows automation only gradually erodes jobs.

Why are there still 90,000 bank branches with over half a million teller and other jobs (just in the US) even after decades of ATMs and Mobile banking? Why do we still have over 600,000 U.S. postal jobs in the face of all kinds of digital communications and when the USPS has automated in the form of kiosks and logistics tech? Why do we still have so many grocery checkout jobs in face of the UPC code/scanner patented 65 years ago and self checkout available for years now? In a world of CGI, why do today’s animated movies show more animators in their credits than Disney’s Snow White did in 1937? How slowly will autonomous cars become mainstream and taxi and truck drivers disappear in a world where half the cars sold globally last year were still manual transmission ?

Their argument – machines are evolving much faster these days so they will destroy jobs much quicker. My counter – technology may be evolving quicker, but technology adoption curves have not speeded up. If anything, my research shows our societies have “circuit-breakers to over automation”. And they go – the past is a poor indicator of the future.

But they don’t have the data. And yet, they want me to be a believer in their lack of Big Data. They want me call their red gray and their yellow white.