It’s the beginning of a new year and we’re being flooded with predictions, retrospectives, and catchy headlines to make us read articles about our favorite topics. One that I have seen more than once is “2015: The Year of Big Data.” I thought 2013 and 2014 were the Years of Big Data! I’m pretty sure that 2016 and 2017 will also be the Years of Big Data.
Click-bait headlines aside (and I’m guilty of it above), Big Data and the infrastructure that supports it and the mathematical analytics that makes sense of it have been producing great descriptions, prescriptions, and optimizations for years. Short of an electromagnetic pulse that will cause us to restart everything someday, Big Data is here to stay. How will things evolve?
First, there will be more data. I take a very Platonic view of data in the sense that whether or not we collect it, it is out there. The amount of sunlight that hits every blade of grass or soybean plant is real, though we don’t measure it all and we don’t therefore have it in some file or database somewhere. We will collect and store more data once we understand why we need it and how to get it.
Second, raw data can be messy and hard to use. We’ll continue to devise ways to clean it up, filter it, compute the missing bits, align overlapping patches of information, and determine metadata that simplifies gaining insights. We’ll figure out more efficient ways of storing the information and invent new kinds of databases. We’ll drastically speed up the time from when we first learn of data’s existence to when we are computing with it and gleaning useful understanding.
Next, we’ll build better and more sophisticated models using this managed data to understand additional kinds of systems and the interconnections among them. We’ll devise more and better algorithms to get more accurate predictions and optimizations, faster. This is the way science works: we build on what we know and have the occasional breakthrough that allows us to do things in new and improved ways or, in some cases, for the very first time.
Finally, this will be translated into more information we can use in practical ways in our personal and professional lives. It will also improve our entertainment and how we read, listen, watch, and engage in sports.
As evidence of how Big Data has been around for a while, sabermetrics, the statistics and analytics of baseball, was being done on computers in the 1960s. We have more data now and better mathematics, but we are seeing an evolution that is expanding and gaining momentum. This is not a statement limited to baseball.
If you are now starting to look into Big Data, good, but get grounded quickly on what it can and cannot do for you. Big Data is just becoming the way we do things. 2015 will be a very good year for Big Data, I agree, but make it great for you and your organization. Whether or not you realize it, it is the basis for how music and movies are recommended to you, how retailers market to you, and increasingly how health treatments are determined for you.
Turn this around and employ Big Data and analytics to make smarter decisions. That doesn’t sound like a new idea for 2015, does it?
This blog entry is also available on LinkedIn.